Although both myself and Greg have been saying (quite publicly) for a long time now that we’re in agreement in about 99% of the DDD/CQRS content we talk about, it turns out the terminology we use has made it very difficult for everybody else to see that.

Anyway, on a recent call with Greg and the Microsoft Patterns & Practices team working on the CQRS guidance, I think we finally ironed out the terminological differences.

First of all, both of us clearly stated that CQRS is not meant to be the top-level architecture of a system.

The use of Bounded Contexts from Domain Driven Design is a good way to *start* handling that top-level.

The area of some contention was how big a Bounded Context should be. After going back and forth a bit, Greg brought the concept of Business Component into the conversation, and that really cleared things up all around. I was quite pleased as I’ve been going on and on about these business components for years (I think 2006 was one of my earlier posts on the topic, though the mp3 has disappeared since then).

Anyway, here’s the meat:

A given Bounded Context should be divided into Business Components, where these Business Components have full UI through DB code, and are ultimately put together in composite UI’s and other physical pipelines to fulfill the system’s functionality.

A Business Component can exist in only one Bounded Context.

CQRS, if it is to be used at all, should be used within a Business Component.

There you have it – terminological agreement in addition to the philosophical agreement that was always there.

You can find the history of my posts mentioning Business Components here.

So, don’t try to change the organization. It’s too big anyway.
Instead, focus on changing one person at a time – that’s hard enough.

Don’t necessarily take the “one person as a time” too literally, though.
You don’t need to completely and utterly have one person won over before starting on the next.

Understand that for someone to change, that may require them admitting (either implicitly or explicitly) that the way they were doing things before was wrong. In some organizations, this can be suicide. Even if it isn’t, psychologically speaking, there are a huge number of barriers to overcome.

So, if at all possible, massage the situation in such a way that it’ll sound like they were right all along, and no-one really understood. It’s easy for someone to play along with the “misunderstood genius” story.

Next time – how to do just that.

Stay tuned.

Andreas Ohlund’s talk on New and Shiny things in NServiceBus 3.0 is available here.

By the way, we’ve now got a beta out of NServiceBus 3.0 – get it here.

Yves Goeleven will be giving a talk on simplifying distributed application development with NServiceBus and the Windows Azure Platform on Jan 31 – details here.

Carl and Richard over at Dot Net Rocks interviewed me at the Oredev conference in Sweden about Domain Driven Design and one of my pet peeves – the use of Customer in example applications. Get it here.

I’m also happy to announce that registration for my 5-day Advanced Distributed Systems Design course has now opened for Bad Ems Germany and New York, in addition to the already open registrations for Austin TX and London. Information and registration on my training page.

I’ve also made some progress with the recording of the course – you can now access days 1, 2, and part of day 3 – covering distributed systems theory, coupling, messaging patterns, bus and broker architectural style, SOA building blocks, and the hotel management SOA exercise. Information and purchase available here.

Many a client seeks me out for the first time for help in this area.

Usually the request is for an amount substantially smaller than infinity.

It’s usually on the discussion groups and in conference presentations that infinity is brought into it.

The basics

The first issue with scalability is the use of the word as an adjective: scalable.

“Is the system scalable?”

Or the similar verb use: “Does it scale?”

The problem here is the implication that there is a yes/no answer to the question.

Scalability is not boolean.

Linear Scalability

When people talk about scalability, or a system being able to scale, they’re usually referring to a graph that looks something like this:

The red graph indicating a system that does not scale well, the green graph indicating one that does.

What is missing from this diagram are the labels of the axes.

The Y axis is Cost, Expense, or Money.
The X axis is usually the number of users (for internet-type companies).

Ultimately, scalability is a cost-function that will tell us how much it will cost to have the system support a certain number of users.

Linear scalability is when the cost of the next user is the same as the cost of the previous user. This means our system doesn’t have bottlenecks. This is what people usually mean when they say “infinite scalability”.

But there’s more

As many of the internet companies (and their investors) have realized over the years, there’s a difference between the number of users and the number of active users. It’s very easy to scale to a billion users when only 1000 of them are active at any given time.

To be more accurate, what we want is additional X-axes for things like total data managed by the system, number of requests per user, resource utilization per request, propagation speed (how quickly information entered by one user needs to be visible to others), and more.

Scalability is a multi-dimensional cost function, where part of an architects job is to figure out which dimensions are significant for the system/business, and what the expectation for growth is across each axis.

Preparing for “infinity”

Be careful not to optimize for only a single dimension – reality is a whole lot more complex.

There are so many other things to deal with as a system scales.

For example, do you really think you’re going to want your configuration entirely centralized? Putting everything in one place means easier management, yes, but it also means a mistake will instantly affect everyone. Is it worth the risk? Maybe instead of centralization, we could do with some automation that will allow a staged rollout of configuration changes with the ability to rollback.

The same goes for rolling out new versions, patches, and upgrades.

But that now means we may have multiple versions of the same system in production at the same time. How will that work? Will they all talk to the same database? How will we version the database then? If not, how will we handle state? Won’t this mean our code will have to be backwards compatible from one version to another? Isn’t that hard? Like, insanely hard?

Please, can we park the whole “infinite scalability” thing?
It’s really not the most important concern – not by a long shot.

Anyway, it turns out that in the Server Core installations of Windows Server 2008 R2, MSMQ does NOT come pre-installed, and for a while it wasn’t even possible to add it (as John Breakwell described here).

Luckily, Microsoft has put MSMQ in the Developer Preview of Server 8 (as Brian Buvinghausen has written here).

Just wanted to call this out in case any of you get stuck on it.

BTW, the training schedule for Q1 of next year is now online, including Germany in February and NYC in March.

Check it out.

At the end of the conference, we were interviewed by the local .NET magazine dNM and that video is now available here. We covered the background on things like DDD, CQRS, and the Cloud. I don’t think that either of us said anything earth-shattering but if you have half an hour, take a look:

It’s not about you.

How’s that for counter-intuitive.

In fact, the more you focus on yourself and trying to get what you want, the more you end up pushing away exactly the thing you’ve been trying to get.

A big part of the reason is:

If what you wanted was entirely under your control, well, you’d already have it.

The fact that you want it indicates that you don’t have full, or often any, control.

Attempting to gain control, as a way of getting what you want, ultimately results in some kind of power struggle (sometimes you may not even realize that this is happening). The thing is that it’s not very smart to engage in a power struggle when your not the one with much control/power.

To the outside world, this attempt may look, at best, futile and at worst, childish – and, if anything, indicates that you really aren’t ready to be given what you want.

Through my work with clients over the years, a common theme has popped up that I haven’t been addressing on this blog up until now. I haven’t yet found a good word to describe this theme, as it is kind of on the nexus of organizational politics, personal development, power, authority, expertise, knowledge, emotional intelligence, and a bunch of other things.

Questions, questions

You see, whenever working with clients, or even when I do my public training courses, when everybody understands my recommendations – what needs to be done, and how that will address the problems they’ve been having – the question invariably pops up: how can we possibly get them (the rest of the organization, our boss, whoever) to let us do this?

I feel that, up until now, I haven’t been giving as much guidance on that side of the equation:

• the “how do I do what I know to be right without getting fired?” question;
• for consultants, the “how do I get my clients to listen to me – to act on my recommendations?” question;
• for managers, the “how do I get my team on board – to follow where I lead?” question

So, in addition to my regular blogging topics I’m going to add the above topics as well.

In short, I’m going to be talking about the answer to the “how do I get what I want?” question.

How do I get what I want?

In the past decade or so, interacting with thousands of people in hundreds of organizations, and in the hundred or so books I’ve digested (and put into practice) on the topic, this seems to be the universal question. During this time, I’ve always shied away from getting into this with clients. It felt out of bounds – off limits, except for the things directly related to the work at hand – even though I knew I could do so much more for them.

In only a handful of cases did I actually go deep, and it was life-altering (for me as well as the client).

While I can’t say that I’ve got it all figured out, in my relatively short time on this planet I’ve managed to get quite a lot of boxes ticked (like finances and living my dream). It’s like I was pre-programmed for a lot of this stuff.

Some things are a never-ending balancing act like “how can I manage all the travel that living my dream entails while still being a loving husband and father?” In my case, it’s 4 little ones – 10, 8, 6, and 10 months, a wife who’s doing her masters in law in parallel to regular work, and me flying out at least twice a month (usually at least a week at a time).

Can’t wait?

For those of you who can’t wait for the blog posts to slowly trickle out, I’m happy to offer one-on-one guidance and coaching similar in form to my regular remote consulting model. If you’re interested contact me at coaching@udidahan.com.

Anyway, I hope this additional aspect to what I do will help you successfully apply all the software things that I’ve been talking about in your current job as well as make the most out of the rest of your career, and your life.

Oh, and the answer to THE question? It’s “just ask”.

I’d really love to hear your comments and thoughts on this additional twist to our ongoing relationship.

Best regards,

– Udi

If you’re on the other side of the Atlantic, you can catch me in Stockholm on Nov 28 or in London on Dec 12.

On a related note, when I gave the course in Australia I had it recorded.

I know that there are some of you who can’t get your companies to give you time off to go to one of my courses so I hope that this virtual training thing will help. It’s quite a lot of video and, it turns out, there aren’t a lot of companies who make it easy to sell videos online. I’ve found one that looks very reasonable, but the visual design side of things still requires a fair bit of tweaking to make it look good.

Regardless, even though it looks ugly, even though not all of the videos are up yet, I wanted to make it available for all of you as quickly as possible. So here it is.

This is my first time doing something like this so please give me feedback on how to improve. Hopefully the platform that I’ve chosen will enable me to incorporate as much of your ideas as possible.

I know some of you wanted to register to my course in San Francisco this October only to see it sell out. There have been a couple of cancellations so there are now some seats available – get ‘em here.

Just as a reminder, when in San Francisco I’ll be giving a presentation to the Bay.NET user group on the evening of Wednesday Oct 26, at 6:30pm. Info and registration here.

In November I’m going to be in Madrid with Eric Evans at the IASA DDD conference, me giving a keynote on CQRS following with a full-day CQRS workshop. After that I’m off to Oredev presenting on CQRS, domain models, and bounded contexts.

Hope to see you at one of these places.

I think I’ve just about drove everybody crazy now with my apparent zigzagging on CQRS.

Some people heard about CQRS first from one of my presentations and got all excited about it. Then I did some blogging which further drove people to CQRS (as did Greg Young and some others). As CQRS was just about to hit its stride with the Early Adopters, I started pushing a more balanced view – CQRS not as an answer, but as one of many questions. More recently I’ve pushed more strongly back against CQRS saying that it should be used rarely.

So what’s the missing piece?

If you’re in the Domain-Driven Design camp (as many doing CQRS are), then it’s Bounded Contexts.

If you’re in the Event-Driven SOA camp (a much smaller camp to be sure), then it’s Services.

The problem is the naming, because the DDD guys have their kinds of services which do not fit the definition for Service of the Event-Driven SOA approach.

Let me propose the term Autonomous Business Component for the purposes of this blog post to describe that thing which is both a DDD Bounded Context (have the shared BC part of the acronym) and an SOA Autonomous Services. Resulting in the nice short form: ABC (and everyone knows you need to have a good acronym if you want something to catch on).

What does this have to do with CQRS?

Nothing just yet. Well, at least, nothing directly to do with CQRS.

Although some proponents of CQRS have stated that it can and should be used as the top-most architectural pattern, both myself and Greg Young (arguably the first two to talk about it and the two who ultimately collaborated on naming it – and now Google knows we didn’t means “cars”) always recommended it as a pattern to be used one level down.

Although Greg and I have had many long discussions on the topic and do agree very much about what the overall structure should look like, I’ll try to avoid putting words in his mouth from this point on.

Before talking more about ABCs, let’s discuss the principle upon which they rest: The Single Responsibility Principle (SRP).

What does SRP have to with CQRS?

Many developers are familiar with SRP and have seen good results from using it. What we’re going to do is take this principle to the next level.

In Object Orientation (OO), data is encapsulated in an object. A good object does not expose its data to other objects to do with as they wish. Rather, it exposes methods that other objects can invoke, and those methods operate on the internal data.

SRP would guide us to not have the same data exist in two objects. For example, if we saw the customer’s first name as an internal data member of two objects, we’d be right to question that kind of duplication and move to refactor it away. However, when we see two systems doing the exact same thing – somehow that gets excused.

“Of course we need to be able to see the customer’s first name in the front-end website as well as in the back-end fulfillment system. How could we NOT have the customer’s first name in both those code-bases?”

And there’s the catch.

Who said that a system should be a single code-base?

But what about integration?

Although many times we do need to integrate existing systems together, sometimes we have the ability to change those systems. More importantly, when going to create a new solution, we can avoid getting ourselves into the problems that integration tries to solve.

Integrating with a system that cannot be changed can be done also by composing multiple ABCs, but that’s a topic for another post.

It is better to think of integration as a necessary evil – kind of like regular expressions and multi-threading; things to be avoided unless absolutely necessary.

“If you have a problem that you decide to use a regular expression to solve, you now have 2 problems.” Or so the saying goes. With multi-threading, you have a non-deterministic number of problems to solve.

If you thought you had duplicate responsibilities with 2 systems operating on the same data, how will introducing a 3rd code base (also known as “integration”) help? Remember that Single Responsibility Principle – our goal is to get it down to one.

OK, so how do ABCs do that?

In order for us to get back into alignment with SRP, that would require us to have responsibility for a single piece of data exist in one code base. Note that SRP makes no statements about how many physical places a given code base can be deployed to. Nor does it state that only a single technology can be in play – code that emits HTML can be packaged at design time together with rich-client code in the same solution.

If an ABC is responsible for a piece of data, it is responsible for it everywhere, and forever. No other ABC should see that data. That data should not travel between ABCs via remote procedure call (RPC) or via publish/subscribe. It is the ultimate level of encapsulation – SRP applied at the highest level of granularity.

This results in systems which are the result of deploying the components of multiple ABCs to the same physical place. The ABC which owns the customer name would have the necessary web code to render it in the e-commerce front-end and in the shipping back-end for printing on labels. This would mean that practically every screen in any UI is a composite of widgets owned by their respective ABCs.

This is ultimately what keeps the complexity of each ABC’s code base to a minimum.

But why not just use CQRS as the top-level pattern? ABCs are weird.

Imagine trying to create a single denormalized view model for the entire Amazon.com product page – product name, price, inventory, editorial review, customer comments, other products that customers viewed, other products that customers bought, etc.

Pretty complex, right?

How much duplication would you have for the page shown after you add an item to a cart? Once again, you need to show other products that customers bought, their names, images, prices, and inventory.

And then on the home page – items you might be interested in, names, images, prices.

And that’s only in the front-end system.

It’s not just the duplication, but how complex the code is for each one.

Instead of the duplication that top-level CQRS would bring you, consider an ABC responsible for products names and images that has just about the same view model composed on each of the above screens. The same with another ABC responsible for price.

You may be thinking that this would result in more queries to get the data to show on a page, and you’d be right. But it isn’t necessarily a classical N+1 Select problem, as the queries are bounded to the number of ABCs. Secondly, consider the ability to have well-tuned caching at the granularity of an ABC – something that would be much more difficult when dealing with everything as a single monolithic view model. In short, not only will it not be a performance problem, often it will actually improve performance.

OK – that explains “everywhere”, what about “forever”?

Forever is where things get interesting – or more accurately, when they get interesting.

Let’s talk about things like invoices.

One of the requirements in this area is that immutability. If the customer’s name was Jane Smith when they made their purchase, it doesn’t matter that they’ve since changed their name to Jane Jones, the invoice should still show Jane Smith.

Often developers push these types of requirements on the data warehouse guys – that’s where history gets handled. The only thing is that if your ABC owns the customer’s name, then no other code base can deal with it. If it’s your data, you have to handle all historical representations of it.

On the one hand, this would seem to kill the data warehouse. On the other hand, it means that the principles of data warehouses are now core to every code-base.

This means you don’t ever delete data (see my previous blog post on the subject), and you definitely don’t overwrite it with an update – even if you think you’re in a simple CRUD domain. The only case where you can get away with traditional CRUD is if we’re talking about private data – data that is only ever acted on by a single actor.

This sounds like the collaboration you talk about with CQRS

It’s similar in principle but different in practice.

In a collaborative domain, an inherent property of the domain is that multiple actors operate in parallel on the same set of data. A reservation system for concerts would be a good example of a collaborative domain – everyone wants the “good seats” (although it might be better call that competitive rather than collaborative, it is effectively the same principle).

A customer’s name would not fall under that category. It isn’t an inherent property of the domain for multiple actors to operate on that data. While there can be multiple readers, one can easily enforce a single writer without any adverse effects. Doing that with a reservation system would cause the online system to behave as if users were lining up in front of a box office – not a desirable outcome.

Private data would be something like a user’s shopping cart. Until they make a purchase, that data doesn’t need to be visible anywhere. Here you could theoretically do simple CRUD – that is, until the business realizes that there’s extremely valuable information to be extracted from the historical record of things people do with their carts.

I think you’re ready to make your point, so just make it already

OK – so we now realize that Update and Delete don’t exist in their traditional form. Delete is really just a kind of update, and update is effectively an “upsert” – a combination of update and insert to retain history. This can be done by having ValidFrom and ValidTo columns for our data.

In which case, Create is really just a special case of Upsert, which looks like this:

UPDATE Something SET ValidTo = NOW() WHERE Id=@Id AND ValidTo = NULL; INSERT INTO Something SET { regular values }, Id=@Id, ValidTo = NULL;

And then we’d have 2 forms of Read – reading the current state (ValidTo = NULL), and reading history (ValidFrom <= Instant AND (ValidTo >= Instant OR ValidTo = NULL))

Here we don’t need fancy N-Tier architectures, data transfer objects, service layers, or domain models. A simple 2-Tier approach could probably suffice. We don’t need a task-based UI, events, denormalized view models, or any of that CQRS stuff. This was at the crux of my previous anti-CQRS post.

The only thing is that this is exactly CQRS.

Say what?

Have we not effectively separated the responsibility of commands/upserts and queries/reads?

As Greg Young has said before, “the creation of 2 objects where there previously was one”.

Effectively 2 paths through our ABC.

CQRS.

Let me give you a second to gather your thoughts.

*

You see, CQRS is an approach, a mind-set – not a cookie cutter solution. Frameworks that guide you to applying CQRS exactly the same way everywhere are taking you in the wrong direction. The fact is that you couldn’t possibly know what your Aggregate Roots were before you figured out how to break your system down into ABCs. Attempting to create commands and events for everything will make you overcomplicate your solution.

So the built-in history of this model is event-sourcing?

Well, it’s not event-sourcing in the sense that we don’t necessarily have events. It achieves many of the benefits of event-sourcing by giving us the full history of what happened.

On the whole issue of replaying events to fix bugs – that’s a bit problematic, logically, unless we have a closed system. A closed system is one that doesn’t interact with anything else – no other systems, no users, nothing. As such, closed systems aren’t that common.

In an open system, one with users, let’s say there was a bug. This bug could have caused the wrong data to be written and/or shown to users. As such, users could have submitted subsequent commands based on that erroneous data that they would not have submitted otherwise. There’s no way for us to know.

The problem with replaying events when we fix the bug is that we’re in essence rewriting history – making it as if the user didn’t see the wrong data. The only problem is that we can’t know which events not to replay – we can’t automatically come up with the right events that should have come afterwards. We could try to sit together with our users and have them try to revise history manually, but our organization often isn’t in a bubble. Our users interacted with customers and suppliers. It isn’t feasible to try to undo the real-world impacts of this situation.

Why didn’t you just tell us this from the very beginning?

I did, you just weren’t listening.

You wanted a cookie cutter, and until you tried CQRS out as cookie cutter (and saw it create a bunch of complexity) you wouldn’t listen to anything else.

As developers, we’re trained to solve problems – the faster the better. Unfortunately, this causes us to be blind to things that don’t immediately present themselves as solutions.

When applying CQRS with ABCs, the solutions you end up with are very simple, but the process of getting there is quite hard and takes practice. Finding the boundaries of ABCs such that data isn’t duplicated between them and that data doesn’t travel between them either via RPC or publish/subscribe – it may feel impossible the first several times you try. Keep at it – it is almost always possible.

We haven’t touched on the whole saga/aggregate-root thing yet, but that isn’t as important until you can successfully apply the principles described here.

Also, this post has already gotten long enough, so it looks like now would be a good time to stop.

Until next time…

For more information, take a look at this video.

You can get this tool on our download page and also via the Visual Studio Gallery here.

The problem

Let’s start with an image to describe the scenario:

The main issue we have here is that the values transaction 2 gets for A and B are those from T0 – before either transaction 1 or 3 completed. The reason this is an issue is that these old values (usually together with some message data) are used to calculate what the new state of C should be.

Traditional optimistic concurrency techniques won’t detect any problem if we don’t touch A or B in transaction 2.

In short, systems today are causing inconsistency.

Some solutions

1. Don’t have transactions which operate on multiple entities (which probably isn’t possible for some of your most important business logic).

2. Turn on multi-version concurrency control – this is called snapshot isolation in MS Sql Server.

Yes, you need to turn it on. It’s off by default.

The good news is that this will stop the writing of inconsistent data to your database.
The bad news is that it will probably cause your system many more exceptions when going to persist.

For those of you who are using transaction messaging with automatic retrying, this will end up as “just” a performance problem (unless you follow the recommendations below). For those of you who are using regular web/wcf services (over tcp/http), you’re “cross cutting” exception management will likely end up discarding all the data submitted in those requests (but since that’s what you’re doing when you run into deadlocks this shouldn’t be news to you).

The solution to the performance issues

Eventual consistency.

Funny isn’t it – all those people who were afraid of eventual consistency got inconsistency instead.

Also, it’s not enough to just have eventual consistency (like between the command and query sides of CQRS). You need to drastically decrease the size of your entities. And the best way of doing that is to partition those entities across multiple business services (also known in DDD lingo as Bounded Contexts) each with its own database.

This is yet another reason why I say that CQRS shouldn’t be the top level architectural breakdown. Very useful within a given business service, yes – though sometimes as small as just some sagas.

Next steps

It may seem unusual that the title of this post implies that SOA is the solution, yet the content clearly states that traditional HTTP-based web services are a problem. Even REST wouldn’t change matters as it doesn’t influence how transactions are managed against a database.

The SOA solution I’m talking about here is the one I’ve spent the last several years blogging about. It’s a different style of SOA which has services stretch up to contain parts of the UI as well as down to contain parts of the database, resulting in a composite UI and multiple databases. This is a drastically different approach than much of the literature on the topic – especially Thomas Erl’s books.

Unfortunately there isn’t a book out there with all of this in it (that I’ve found), and I’m afraid that with my schedule (and family) writing a book is pretty much out of the question. Let’s face it – I’m barely finding time to blog.

The one thing I’m trying to do more of is provide training on these topics. I’ve just finished a course in London, doing another this week in Aarhus Denmark, and another next month in San Francisco (which is now sold out). The next openings this year will be in Stockholm, London; Sydney Australia and Austin Texas will be coming in January of next year. I’ll be coming over to the US more next year so if you missed San Francisco, keep an eye out.

I wish there was more I could do, but I’m only one guy.

Hmm, maybe it’s time to change that.

Info and registration here.

Hope to see you there.

You see, the majority of our focus since releasing version 2.5 at the end of last year was to building our amazing new version 3.0 with all of its slick new coolness. But then we started getting requests from companies here and there using 2.5 to improve this small piece, that small piece, etc.

For a while, we thought we’d get version 3.0 out the door fast enough to address it all in one fell swoop. It turns out we couldn’t – version 3.0 had too many big changes in it that would take too long to stabilize. And thus, version 2.6 was born.

Get it here.

Customer Testimonial

I’m also thrilled to announce that we got permission to go public about ****** one of the world’s largest investment banks using NServiceBus (publication pending final corporate approval). They were kind enough to provide us with this testimonial:

NServiceBus is an exceptional framework for rapidly building enterprise applications. Not only was it easy to use out of the box, but the documentation was great and the large user community was very responsive. We were up and running with a prototype in a week.

We turned to NServiceBus for its performance and scalability for our new Structured Notes publishing systems. The reliability it provided us in integrating with other systems like Exchanges, IPA and code providers, as well as our internal systems allowed our developers to deliver on our core business objectives very quickly.

Like most large enterprises, we need full control of all of the internal behaviors of our infrastructure. We found the multiple layers of extensibility in NServiceBus gave us the control and flexibility we needed without sacrificing stability.

A truly exceptional framework.

– Manoj Oswal, VP – Structured Notes, Tier 1 Investment Bank

And in other news…

We’re hiring!

Business is booming and we’re looking to grow, currently focusing on the business side of the company. This includes project managers, practice managers, and program managers as well as sales and marketing people.

I’m also looking for a Chief Operating Officer who will be responsible for this group.

If you’re interested – please let us know: recruiting@nservicebus.com

Our HR story isn’t yet very well developed so please bear with us as we ramp up.

What’s RavenDB?

For those of you who haven’t heard about RavenDB yet – it’s a transactional document database for .NET, and it’s been/being developed by my good friend and partner in crime Ayende Rahien (a.k.a Oren Eini). Although Ayende is known for his “not invented here” tendencies, when it comes to transactional document databases, well, you’d have been hard pressed to find one – especially with a decent .NET API.

NServiceBus Storage

In NServiceBus we have a variety of storage needs – from things like durable subscriptions so that you get fault-tolerant pub/sub, through persistence of long-running workflow (a.k.a saga) state, and durable timeouts so that your time-bound long-running processes never get stuck. None of these actually require relations – we were just using relational databases for storage because it was the easy answer everyone was going with.

Relational vs. Document

And these relational databases came with some downsides – developers had to “beg” their DBA to create the needed tables in the production databases, when that central database was down it prevented otherwise autonomous publishers and subscribers from going about their business, and it was very difficult to version the long-running workflows especially when newer versions required different state.

By moving to an embedded and transactional document DB, no longer do you need to bargain with the DBA, by keeping the storage together with the processing nodes you are back in parallel processing bliss, and the schema-less nature of the storage makes versioning those long-running workflows much easier.

Licensing

And I’m also happy to announce that an agreement has been reached with Hibernating Rhinos (Ayende’s company) so you won’t need to license RavenDB separately to get all of these benefits. RavenDB will be bundled with NServiceBus so licensing NServiceBus covers them both (for the storage needs mentioned above).

If you’re also thinking about using RavenDB as the storage for the rest of your system, you’ll be able to get a nice discount on the license when purchasing it together with NServiceBus. This will be going into effect with the release of NServiceBus 3.0 this October.

Companies Using NServiceBus

Interestingly enough, there’s been a big uptick in companies using NServiceBus with the introduction of licensing. Most of these companies are not the kind of big, stand-up-and-take-notice names that everybody likes to have on their roster.

What makes NServiceBus particularly attractive is that you can get a lot done without requiring some kind of dedicated BizTalk/WebSphere/Tibco expert. This has brought down the barrier for thousands of developers who just want to get on with the business of getting their app to market.

And when it comes to the big names, well, once they see how much faster they can get stuff done with NServiceBus as well as how robust and scalable it is in production, they don’t want any of their competitors to know about it!

Anyway, I’m glad to say that two companies have stepped forwards: Rackspace and Reuters. Hopefully we’ll get confirmation from one of the big banks soon that we can go public with them too.

Exciting times ahead.

I’m also giving the course this year in Oslo, Norway; London, UK; Aarhus, Denmak; and Stockholm, Sweden. Dates and registration available here.

If this is the first you’re hearing about my course, you can find out more about it here.

The question

The main touted benefit of these workflow-centric architectures is that we don’t have to change the code of the system in order to change its behavior resulting in ultimate flexibility!

Some of you may have already gone down this path and are shaking your heads remembering how your particular road to hell was paved with the exact same good intentions.

Let me explain why these things tend to go horribly wrong.

What’s behind the curtain

It starts with the very nature of workflow – a flow chart, is procedural in nature. First do this, then that, if this, then that, etc. As we’ve experienced first hand in our industry, procedural programming is fine for smaller problems but isn’t powerful enough to handle larger problems. That’s why we’ve come up with object-oriented programming.

I have yet to see an object-oriented workflow drag-and-drop engine. Yes, it works great for simple demo-ware apps. But if you try to through your most complex and volatile business logic at it, it will become a big tangled ball of spaghetti – just like if you were using text rather than pictures to code it.

And that’s one of the fundamental fallacies about these tools – you are still writing code. The fact that it doesn’t look like the rest of your code doesn’t change that fact. Changing the definition of your workflow in the tool IS changing your code.

On productivity

Sometimes people mention how much more productive it would be to use these tools than to write the code “by hand”. Occasionally I hear about an attempt to have “the business” use these tools to change the workflows themselves – without the involvement of developers (”imagine how much faster we could go without those pesky developers!”).

For those of us who have experienced this first-hand, we know that’s all wrong.

If “the business” is changing the workflows without developer involvement, invariably something breaks, and then they don’t know what to do. They haven’t been trained to think the way that developers have – they don’t really know how to debug. So the developers are brought back in anyway and from that point on, the business is once again giving requirements and the devs are the one implementing it.

Now when it comes to developer productivity, I can tell you that the keyboard is at least 10x more productive than the mouse. I can bang out an if statement in code much faster than draggy-dropping a diamond on the canvas, and two other activities for each side of the clause.

On maintainability

Sometimes the visualization of the workflow is presented as being much more maintainable than “regular code”.

When these workflows get to be to big/nested/reused, it ends up looking like the wiring diagram of an Intel chip (or worse). Check out the following diagram taken from the DailyWTF on a customer friendly system:

The bigger these get, the less maintainable they are.

Now, some would push back on this saying that a method with 10,000 lines of code in it may be just as bad, if not worse. The thing is that these workflow tools guide developers down a path where it is very likely to end up with big, monolithic, procedural, nested code. When working in real code, we know we need to take responsibility for the cleanliness of our code using object-orientation, patterns, etc and refactoring things when they get too messy.

Here is where I’d bring up the SOA/pub-sub approach as an alternative – there is no longer this idea of a centralized anything. You have small pieces of code, each encapsulating a single business responsibility, working in concert with each other – reacting to each others events.

Productivity take 2: testing and version control

If you’re going to take your most complex and volatile business logic and put it into these workflow tools, have you thought about how your going to test it? How do you know that it works correctly? It tends to be VERY difficult to unit-test these kinds of workflows.

When a developer is implementing a change request, how do they know what other workflows might have been broken? Do they have to manually go through each and every scenario in the system to find out? How’s that for productivity?

Assuming something did break and the developer wants to see a diff – what’s different in the new workflow from the old one, what would that look like? When working with a team, the ability to diff and merge code is at the base of the overall team productivity.

What would happen to your team if you couldn’t diff or merge code anymore?
In this day and age, it should be considered irresponsible to develop without these version control basics.

In closing

There are some cases where these tools might make sense, but those tend to be much more rare than you’d expect (and there are usually better alternatives anyway). Regardless, the architectural analysis should start without the assumption of centralized workflow, database, or centralized anything for that matter.

If someone tries to push one of these tools/architectures on you, don’t walk away – run!

I can only speak for myself on this one, but I guess it’s that there’s just so many times you can repeat yourself.

So, why this post?

Well, Richard was able to dig up an old (2004) presentation I gave about SOA in which I said:

“Services run in a separate process from their clients
A boundary must be crossed to get from the client to the service – network, security, …”

And 7 years later I can say, hand on heart, I was wrong.

Luckily, I’ve spent much of those past 7 years trying to correct that recommendation. One blog post in which I tried to do that (in mid-2007) was On Intermediation and SOA in which I described the relationship between systems (i.e process boundaries) and services:

“all of these “systems” might just end up within the same service, or having parts of them being used by multiple services

There can also be multiple services (or, more accurately, parts of multiple services) deployed together in the same system/process.

And this is nothing new – in the 4+1 Architectural View Model by Philippe Kruchten (1995) we can see very clearly the differentiation between the Logical View (our services) and the Physical View (a.k.a the Deployment View).

These views are orthogonal to each other – multiple elements from one view can map to a single element in another view (and vice versa).

This, if anything, makes it that much harder to identify service boundaries – if they have nothing to do with the existing applications and systems, then what are they? In my blog post on The Known Unknowns of SOA I point to the fact that Business Capabilities are much more appropriate constructs than, say, web services which (as it says in the referenced post) “[are] merely a standardized approach to accessing functionality on remote systems”.

As I bring this post to a close, I’m feeling more comfortable rehashing material I’ve published before:

Logical and Physical Architecture

and the rest of the SOA category on my blog here.

Happy boundary hunting.

NServiceBus on DNR-TV

For some more advanced stuff, I suggest looking at the Hidden NServiceBus Gems talk that I gave at Skills Matter the week after. Here we get into all sorts of things that you won’t tend to find by yourself through regular usage of NServiceBus.

Hidden NServiceBus Gems

Enjoy.

I’ll also be giving a presentation at YOW Nights on July 5th. Information here.

I’ve also decided to give a condensed version of my 3-day Enterprise Development with NServiceBus course while I’m there. This “NServiceBus Kickstart” event will be on Saturday July 9th and will give you everything you need to know to get started with NServiceBus.

Joining me in leading this event will be Simon Segal and Mark Harris – two of the earliest adopters of NServiceBus in the world. They’ve built a very impressive product with NServiceBus that has been in production with the Australian Federal Government for over 12 months now and will be sharing from their experiences.

Register here.

Since that post I’ve blogged about many techniques and approaches to identify better boundaries (like with SOA and DDD) and I’m seeing more and more developers starting to apply them.

This post will be slightly different.

You see, occasionally we technical people will get requirements that can’t easily be broken down by functional boundaries. Sometimes the business calls this a “platform” – here’s an example:

“We want a flexible, customizable workflow-driven platform that allows end-users to add their own columns to any screen able to support massive datasets for large enterprise customers that will also be intuitive and easy to use for our SaaS push to small and medium-sized businesses (SMBs) – oh, and then it needs to be multi-tenant too. Did I mention that we promised this would be ready for our most important client by the end of the year?”

There is only one reasonable answer to the above:

“I know you want it but, I’m very sorry, you can’t have that. It isn’t possible to do that with one system.”

It’s really rare for a technical person to say something like that. The simple reason is that in software, we believe that almost everything is fundamentally possible – given enough time/money/resources. So, when someone in business comes to us with the requirements above, we say “It’s *possible*”, loosely translated to “We can’t prove that that’s impossible”.

There are many reasons why you can’t be SaaS and Enterprise at the same time – not all of them dealing with software. The marketing, sales, and support stories of each of those markets is VERY different. In the enterprise you’re usually working with professional services people that customize the generic product – which then becomes a backwards-compatibility requirement for new development. This will hinder the development team’s ability to roll out the new shiny features needed to remain competitive in the SaaS space.

In the cases where I’ve been brought in to help clients with these kinds of systems, I try to work my way up to the person in management who is in charge of the project/product – often the CEO. Then, in the nicest way possible, I explain that really the only way to have your cake and eat it too is to create 2 companies – each one focused on its own space – Saas and Enterprise; each one with its own development team, feature set, release schedule, etc.

There’s a reason that there’s an SAP *and* a Salesforce – it’s because no one can be all things to all people. It’s hard enough to become a market leader in just one space. Trying to do both is VERY expensive, and increases the chance of the project failing from the average 60-70% in the industry to probably about 99.9%.

Hope that will save you some grief.

Domain Models & Composite Applications

Enjoy!

I’m happy to announce that I’ll be giving my 5-day course in Aarhus Denmark on September 19th. This is in addition to June 13 in London; July 4 in Sydney; and August 15 in Oslo.

I’ll be coming back to the US on Oct 10 to give the course in the San Francisco area, but don’t have a venue yet. Contact me if you can recommend a good location.

Also, Andreas will be teaching my 3-day NServiceBus course in London on June 1st with Skills Matter – you can register here. He’s also giving an “in the brain” user-group presentation on June 2nd on reliable integrations with NServiceBus.

I also forgot to mention that Adam Fyles created some awesome Visual Studio templates to speed your use of NServiceBus – get them here.