I got this question submitted the other day on an old post from 2007 on autonomous services and enterprise entity aggregation:

What happens when two services must be able to create/update the same entities?

For example, take this scenario, lets suppose we have two services: Marketing Service and Product Inventory Service. Marketing Service must be able to create new instances of the business entity Products, as the marketing team needs to make publicity about new products before the products is available for sale. On the other hand, the marketing department will not make publicity for the all products, so the Product Inventory Service needs to be able to create new instances of the Product too.

How to implement autonomous services when there is more than one service which is authoritative services over the same business entity?

Where confusion starts

When people look to allocate responsibility to their services, there are a number of implicit assumptions that are often made that make it difficult to follow the rules of service design.

Problematic Assumption #1

Services should be aligned with organizational boundaries.

For the full story as to why this isn’t a good idea, read my post on people, politics, and the single responsibility principle.

In the question above, assuming that the Marketing Service does all the things that the Marketing Department does, and inversely doesn’t do the things the department doesn’t do gets us into trouble. If a service is responsible for the creation of an entity, there shouldn’t be any other service that can create that entity. So, in this case, the “Marketing Service” (assuming that that would still be the best name for it) would always create the “Product” entity – assuming that that is the right name and set of attributes to put together.

Problematic Assumption #2

Entities should represent “real world” things.

For the full story as to why this isn’t a good idea, read my post don’t try to model the real world, it doesn’t exist.

In the question above, assuming that there is an entity called Product and that entity owns all data about a product like its’ name, price, and amount in inventory gets us into trouble. You see, each of those attributes relates to very different business processes when we look at their use in transactional logic. I mean, sure, you may need to show all the data on one screen in the UI, but that can be done fairly simply using UI composition techniques. For some more advanced UI composition ideas, see the video in this post on service-oriented composition.

Anyway, instead of having a single Product entity, we could have a “CatalogItem” entity that would encompass information about the “product” like its name, description, image, and category information. That entity would share a common ID with entities owned by other services – like an “InventoryItem” that would own information like the quantity of the product.

Depending on other more detailed information about the domain, we may decide on whether data like the dimensions and weight of the product would go together with the quantity or on a separate “ShippableItem” entity owned by yet a different service. One justification for the separation is that the quantity of inventory is quite a bit more volatile than the dimensions and weight. Still, we could have separate entities for those things in the same service.

And that’s why it’s so difficult

When you can no longer rely on the crutches of organizational structure and the nouns of the domain, it becomes much harder to model your domain – dividing up responsibility into services and setting up the right entity model for each of them. At that point, you really need to dive deep into the domain and analyze the way each attribute of data is used and what kind of transactional integrity constraints are real and which are imagined.

For those of you who haven’t gone back and read all of my older posts (and let’s be honest, nobody does that when they subscribe to a blog), I hope the links above give you some next steps to take in learning about service design.

If you want even more

I’ve been teaching about this stuff for a while now and most of the people who’ve attended the training think it’s quite helpful. To get a sense of what this course is like, check out this short video

The thing is that these courses fill up pretty quick – the upcoming ones in Dallas Texas in March and London UK in April have already sold out, so I’m going to try to do more this year than just the four I did last year.

The next one that is now open for registration will be in Sydney Australia in May.

I’m going to try and see if I can get back to the US in August and London in December, but for now the next other course that is already available will be in Denver CO in November.

For those of you who would have a hard time traveling or taking time off work, you can get access to the recording of the first two days of the course – totally free.

More questions?

Feel free to leave a comment and I’ll do my best to help you out.

Seeing as it’s been just about a year since the last NSBCon we ran in New York City, I’m happy to say we’re bringing NSBCon back to the US – this time in Dallas Texas, Dec 1-4.

Click here for the full details.

First – the war stories

For those of you who have been wondering about what it would be like to apply the Service-Oriented principles I blog about, we’ll hear the war story of one brave architect’s journey down that path, which kinds of messaging patterns helped, and which it didn’t.

Another point of concern for people considering event-driven approaches to building systems is how to keep control of all of these loosely-coupled bits and pieces. Luckily we’ll have a session talking about various monitoring and devops techniques to keep on top of everything that’s happening – and not just a theory of how that could be done, but a “from the trenches” account of how well they actually worked.

Then – down to the metal

In addition to some really “down and dirty” sessions about all the cool nifty NServiceBus features that have come out this past year, we’ll also give you a sneak peek into what’s coming in the future (after keeping very tight lipped about our future roadmap).

And finally, we’ll talk a bit about the actor model and Akka.net – where they fit in message-driven architectures and how to get the two worlds to play together nicely.

And finally – an unconference

When all is said and done, what you really want is some time to talk through your specific challenges with people who’ve “been there” and “done that”. That’s why our last day is a full interactive “unconference” meaning that you get to set the agenda, get direct access to all the speakers, and even grab some NServiceBus experts to debug some code with you.

If you’ve never been to an unconference before, you don’t know what you’re missing.

So, hurry up!

Discounted “early bird” tickets are almost sold out.

I really hope to see you all there.

Check it out.

Well, that wasn’t it

Now, even though I did scale back the training quite a lot – from around 10 public courses a year down to 2, they didn’t stop entirely. After settling into what appears to be a new stable and sustainable rhythm, I’m experimenting with bringing the training up to 4 times a year / once a quarter. So far, so good.

Beyond a tweet here and there about those courses, I haven’t been blogging about them – or blogging much at all really (this will be my 3rd post this year).

Anyway, I’m getting back into the swing of things and there are some courses coming up:

Next up Sydney, then Dallas

The next one is in Sydney Australia Oct 12-16 – register here. Somewhat surprisingly, this is pretty close to selling out so if you’re down under, you’ll probably want to sign up for this one as I probably won’t be back for at least a year.

The one after that will be in Dallas TX, March 7-11 – register here. I know there were a lot of you in the US that wanted to go to the last one in Austin, but that one sold out pretty quick. Hopefully you’ll be able to get in this one – and there’s still plenty of time if you want to get the early-bird price.

Looking forwards

Other than the courses, you can find me (and a bunch of other people in Particular) speaking at various user groups and conferences around the world. Here’s the list of upcoming events. Come up and say hello

* Here’s a gist of some ugly javascript to get the source data for that.

In Closing

I really do enjoy teaching and seeing how people’s eyes light up when something all of sudden clicks for them. I’ll probably continue teaching, in one form or another, for quite some time.

Hope to see you at one of my next courses or events.

This one is from µCon 2014: The Microservices Conference at Skills Matter in London.

The title of this talk was: An Integrated Services Approach

and the description:

After many years of the largely enterprise-scale SOA philosophy being applied across multiple systems, we’re now seeing some of that philosophy being applied to the design of the systems themselves with Microservices. Unfortunately, unless we integrate these enterprise and system level philosophies appropriately, we’ll end up with a mess of data duplication and coupling that may even result in businesses running on inconsistent data. Join Udi for a discussion of a unified approach that leverages the best of both worlds.

Hope you find it interesting.

If you want to learn more about these topics, check out more of my posts on SOA here.

If you want the full, in-depth, zero-to-sixty experience – you should really attend my Advanced Distributed Systems Design course. The next one is in March in San Francisco but there will be others around the world through the rest of this year.

For the full list of events, click here.

For those of you who aren’t familiar with the format, an “Ask Me Anything” (or AMA for short) is a kind of online “press conference” which was popularized on Reddit, and has had fairly well-known people do them including President Obama, Bill Gates, Madonna, and more. For more information, see Wikipedia on the topic.

Here’s how it works

First thing, sign up at http://particular.net/ask-udi-dahan-anything to get the conference info.

Then, check out and vote on the questions people have already asked, and ask your own.

Of course, you can also loop some friends in to vote for your questions, but I will do my best to answer as many questions as I can – live on Thursday Dec 11, at 12pm EST (5pm GMT).

Even if you can’t make that time, the recording will be available afterwards. Once I have the link for that, I’ll blog it as well.

But don’t just take a passive wait-and-see approach. If you’ve been reading my blog, I know you’ve got questions about SOA, DDD, CQRS, messaging, NoSQL databases, and more – so this is your chance to get them answered.

Go on then: Ask me anything.

Superficial Invariants

Taken from the blog post mentioned above: “For example, an employee cannot take more annual leave than they have.”

This falls into the trap of applying mathematical thinking (which we developers possess in great quantities) to the business world. The business world isn’t that mathematical (in general), and tends to have many more shades of gray expressed as “business rules” which can, and do, change.

Rules – not invariants

Employees can’t take more annual leave than they have.
… unless their manager approves.
… but that’s only up to 2 days.
… unless their manager is a VP, and then it’s up to 5 days.
… and that negative balance will be deducted from next year’s leave.
… Oh, and if the employee leaves the company before then, then the value of those negative days will be deducted from their final paycheck.

Impact on your domain

First of all, I hope you see that this isn’t something that you would trivially implement on an Employee object.

If you read these rules more carefully, you’ll probably notice that they’re speaking about a long-running process.

First, there is a request for leave. Then there’s an approval (with certain rules) which may come sometime later. And the approval itself may not even end the process – if the balance becomes negative.

And, as you’ve probably heard me say before, you end up with sagas as your aggregate roots (see Race conditions don’t exist from 4 years ago).

And a word about Bounded Contexts

Notice that these rules don’t care very much about things like the employee’s name, phone number, email, etc. Similarly, logic that deals with that data probably doesn’t care about the number of days of leave an employee takes.

In other words, these sets of data and logic can be said to belong to different Sub-Domains (in DDD terminology).

As such, it can make sense to take the annual leave logic and put it in a bounded context separate from the one responsible for the contact info.

In closing

In many of the samples and blog posts I see online, an overly simplified problem domain is implemented showing how the given implementation technique would be applied.

The problem is that developers then use that implementation technique as a “cookie cutter”, trying to fit real-world requirements into it, and then end up making a pretty big mess.

The more you delve into real-world requirements of business domains, the less you’ll see of mathematical invariants (unless, maybe, you’re building a physics engine for a game or something) and the more you’ll see long-running processes unfolding in front of you.

Regardless of whether you use NServiceBus sagas or not, start looking at the world as dynamic long-running processes rather than static noun-centric entities.

Wegmans have been long-time customers of NServiceBus and I recently got the opportunity to go on a guided tour in their flagship store in Rochester and hear all the ways that they’ve been leveraging our platform – even the scales that people use to weigh their produce are linked up.

Yup – you may even be eating NServiceBus and not know it

There are also some pretty powerful back-office processes at play – things that your regular consumer wouldn’t even notice but that matter quite a bit.

In any case, I’ll hand this over to Adam Fyles – the person who has done the actual work of getting NServiceBus into Wegmans, from the early proof-of-concept days to leveraging it across the enterprise to tell the story himself.

Here’s Adam presenting at NSBCon USA 2014:

And if you want the slides or to hear more from Adam, check out his blog at: http://adamfyles.blogspot.com/.

For more information on NServiceBus and the rest of the Particular Service Platform, check out http://particular.net/

The venue was packed, the speakers were great, and all in all the conference was a smashing success. If you don’t believe me <wink/> you can check out Roy Cornelissen’s coverage here.

All videos now online

For those of you who couldn’t make it, we have all the videos recorded for you and you can access them here.

Some of the cooler demos included:

• Szymon Pobiega showing how he plugged the Event Store under NServiceBus for complete transport, subscription, timeout, and saga persistence – video here
• Dylan Beattie videoing the audience with his iPhone and having that processed through NServiceBus and uploaded to their site in “real time” – check it out

Of course, I really think you should check out the full list.

Onward – to New York

And for those of you in the US and Canada, we’re happy to announce that registration is open for NSBCon New York!

The early bird is available until the end of this week (Aug 8), so you should really get going.

Register here.

We’ve got some really great speakers lined up for you – Jimmy Bogard will be there, and we’ll have both Oren Eini and Ted Neward speaking as well (who you might remember from the infamous ORM Smackdown).

And Carl and Richard from Dot Net Rocks will be there as well – rockin’ it the way that only DNR can.

“Doesn’t all of this loosely-coupled composition come with a high cost, in terms of client to server chit-chat?”

So, I’ve finally buckled down and put together a slide to illustrate how the technocratic IT/Ops service I’ve talked about in the past can provide components to resolve these sorts of problems.

After putting the slide together, and realizing some animation would do it good, I went and made a short (5 min) video including some verbal explanation as to how it all works – just for clarity. Check it out or watch it here:

And here’s the image showing everything in one picture:

For those of you who have been wondering, yes, you can come to just one of the days if you aren’t able to take both days off of work. Also, if you want to come but can’t convince your boss to pay for it, drop us a line and we’ll hook you up with a discount. Just send an email to nsbcon@particular.net and we’ll take care of everything.

One last thing – unfortunately, Oren won’t be able to come speak in London (but still will be speaking in NYC). While no one can quite fill in those shoes (as Oren is quite a big guy), we have someone I think you’ll really enjoy hearing: James Lewis, one of the main voices in the Microservices arena, will be coming to talk about … Microservices!

Hope to see you there: NSBcon.com.

When you write a software module, you want to make sure that when changes are requested, those changes can only originate from a single person, or rather, a single tightly coupled group of people representing a single narrowly defined business function. You want to isolate your modules from the complexities of the organization as a whole, and design your systems such that each module is responsible (responds to) the needs of just that one business function.

This is something that often comes up when I teach people about service boundaries when it comes to SOA – organization boundaries are the most intuitive choice.

And, once up on a time, that intuition might have indeed held up.

Stepping back in time

In the age before computers, organizations had a very specific way of structuring themselves.

People who had to work closely together sat in close physical proximity to each other. Data that was required on an ongoing basis would be in file cabinets also physically co-located with the people using that data, and it would be structured in a way that was optimal for their specific purposes. All of this was due to the high cost of communicating with people farther away.

If you needed data from a different department, you had requisition it by filling out a special form, put it in your outbox, and then some guy from the mail room would pick it up, and physically schlep it to the right department, putting it in their inbox, and then someone there would get your data for you – putting it together with your original request, and then the mail guy would schlep it back. This inbox/outbox style of communication should ring a bell from the messaging patterns I talk about with NServiceBus.

As a result, different departments had to have very clearly delineated responsibilities with minimal overlap with each other. The organization just couldn’t function any other way.

And then a bunch of us geeks came along.

Enter the age of computers and networks

By introducing this technology, the cost of communication across large distances started falling – slowly at first, and then quite dramatically.

When anyone in an organization was able access data from anywhere in the blink of an eye, an interesting dynamic started to unfold. All of a sudden, the division of responsibility between departments wasn’t as critical as it was before. When an employee needed to do something, there wasn’t this “that isn’t our job, you need to go to so-and-so” reaction. Because things could be done instantly, that’s exactly what happened.

And then came the politics

By removing the cost of communication, it became possible for more power-hungry people in the organization to start making (or trying to make) decisions that they couldn’t have made before. The introduction of computers into an organization was heralded as a new way of doing business – that the old organizational boundaries were a relic that we should leave behind us.

And thus can the re-org (the first of many).

Responsibilities and people were shuffled around, managers vied for more power, and politics took its’ place as one of the driving forces in the company structure.

Nowadays, if you want a decision made in a company, there isn’t just one person who has the authority to sign off on it anymore. No, you need to have meetings – and more meetings, with people you never knew existed in the company, or why on earth they should have a say on how something is supposed to get done. But that is now our reality: endlessly partially overlapping responsibilities across the organization.

So, what of the Single Responsibility Principle

This just makes it that much harder to decide how to structure our software – there is no map with nice clean borders. We need to be able to see past the organizational dysfunction around us, possibly looking for how the company might have worked 100 years ago if everything was done by paper. While this might be possible in domains that have been around that long (like banking, shipping, etc) but even there, given the networked world we now live in, things that used to be done entirely within a single company are now spread across many different entities taking part in transnational value networks.

In short – it’s freakin’ hard.

But it’s still important.

Just don’t buy too deeply into the idea that by getting the responsibilities of your software right, that you will somehow reduce the impact that all of that business dysfunction has on you as a software developer. Part of the maturation process for a company is cleaning up its’ business processes in parallel to cleaning up its’ software processes.

The good news is that you’ll always have a job

Just as an aside, I’m not currently planning on running another course this year so you might want to try to make this one before it sells out.

In any case, here’s the description of the presentation:

Ever since callbacks made their way into programming languages, developers have been arguing whether the benefits of looser-coupling outweigh the drawback of seeing the control flow in one place.

These arguments continue as developers look to leverage technologies like RabbitMQ on-premise or Azure Queues and Amazon SQS in the cloud. See which development practices can bring the global flow visualization of orchestration to your publish/subscribe messaging architectures to get the best of both worlds.

The presentation will be at Microsoft, 11 Times Square (map)
Room 6501a, 6th Floor. Make sure you bring a photo ID for building security.

Register here.

Hope to see you there!

The speaker lineup for NSBcon London 2014 is now complete and we’ve got a really great mix of talks, if I do say so myself.

I’ve already mentioned that Oren and Greg will be there, but I wanted to talk a bit about the rest of the roster:

First up – Wonga

If you’re living in the UK, you almost can’t avoid seeing these ads.

What you probably didn’t know is that Wonga has been running on NServiceBus for years now.
(No – we had nothing to do with the ads, and there’s nothing we can do about them.)

Charlie Barker was there from the beginning and has lived to tell the tale:

We faced the problem of scaling our platform to meet rapid growth in customers whilst at the same time increasing the team from 15 to 200 people spread across five countries. We did this by transitioning from N-Tier to SOA even as we were delivering new features and keeping the platform stable. No small feat.

This is an interesting story to hear, not only from the point of view of NServiceBus, and I’d definitely recommend grabbing Charlie over lunch or over a beer. This is probably one of the higher profile startup successes in the UK you’ll find and, as always, the behind the scenes story is just fascinating.

And to the cloud!

No self-respecting technology conference these days can go without spending at least some time talking about the cloud – and we won’t be the ones to buck the trend, even though we have a healthy disrespect for all sorts of things, including the ourselves

You’ll hear from one of the foremost Azure MVPs and all around cloudy Belgian Yves Goeleven (whose last name nobody is really sure how to pronounce). Yves has been the driving force behind getting all the various bits of Azure infrastructure integrated into NServiceBus and, if you get him to just the right level of inebriation, will spill all the dirty little secrets of the Azure platform that Microsoft doesn’t want anyone to know.

Dylan Beattie will then relate his tales of creating loosely-coupled encoding workflows for audio and video on the cloud at Spotlight – one of the world’s leading resources for professional actors, casting directors, and production professionals. It ain’t easy making ordinary people into stars.

And so much more

To see the complete lineup, go to NSBcon.com.

And me – what will I be talking about? That’s a good question.

Not to steal my own thunder (after stealing everybody else’s), but you’ll hear about the deeper integration we’ve got planned for SignalR, making your event-driven architecture extend from the back of your systems all the way to the browser, so much simpler and smoother than you ever thought possible.

I’ll also tell you about how we’re going to enable you to run on queues that don’t support distributed transactions (like Service Bus for Windows Server) without having to worry about making your logic idempotent. For some background, see my blog post on Life without distributed transactions.

In short – it’s going to be a kick-ass conference.

Check it out.

* and for those of you paying out of pocket, contact nsbcon@particular.net for a discount.

This came up when I was in London for the DevWeek conference at the speaker dinner with Austin Bingham, Rob Smallshire, and Allen Holub and I wanted to share some of the ideas that came up, as well as some of my personal opinions on the matter.

Context matters

When you’re in an organization that is continuously developing and evolving a product, platform, or suite, your context is quite different than when you’re working on a project either for an external client or an internal one.

In short: product vs. project.

While in both contexts you’ll want frequent releases, the main difference is that a project is meant to achieve a certain state of “done-ness” in a bound period of time. A product is not ever meant to be “done” in that way. A successful product is one that continues to evolve over time, with that success (arguably) resulting in more resources being dedicated to its development.

If we were to zoom out our scope beyond that of the project, we’d probably see certain product-ish qualities at the level of a client’s entire IT environment – no state of “done-ness” and similar consequences of success.

Where estimates are needed

Whether you’re in a product development organization or the CIO of an enterprise, there are a certain number of features/projects (FP) that people in “the business” want done. Let’s assume that each FP has a certain amount of business value that its implementation would result in and that that value is known in advance.

Sidebar: Clearly, the business value of any feature or project can not be known with much certainty in advance of it being implemented. Still, for the purpose of keeping the analysis simple for now, let’s table this issue for a bit.

While you might think it’s reasonable to perform the work on these FPs in order of decreasing value, that is mistaking revenue for profit/return-on-investment (ROI).

In other words, we need to know roughly how much each of them costs to be able to calculate its predicted ROI (value – cost).

Only then can we decide in which order to do the work.

So, we need somebody technical to give an estimate.

High-level estimates

In this context, sometimes it’s enough to provide 3-4 buckets describing the amount of work – I like the approach of using shirt sizes: S, M, L, XL.

This can help the organization decide quickly to charter the development of the FPs with very high value and very low cost. These low hanging fruit are great for getting started, but when you’re done with them, then you have to decide between a bunch of FPs whose predicted ROI are all very close to each other.

Before doing that (!), it’s important that no projects with an XL size are fully chartered as is – no matter what the value.

When a technical person gives an XL estimate, that means “this is so big, I really have no idea how long it’ll take”. The variance can be huge here – in some cases, they might not even be certain if the request is doable without being given some time to do additional research first. And that is exactly why you need to carve out a certain chunk of time and resources for doing that research.

“But you don’t understand, Udi! We need this done ASAP.”

Believe me when I say that that ship has already sailed.

There’s a pretty good chance that before that extra-large FP is half-done, so much time will have gone by that business priorities will have shifted. Unfortunately, by that time so many resources will have been invested in it that nobody will have the guts (or political capital) to pull the plug on it. The best people will start leaving – sometimes to other FPs but, more often, the company as well. Even when these nightmarishly large missions are eventually done – they end up being something of a Pyrrhic victory.

So, what to do?

Well, beyond financing a certain amount of directed and scoped research and development to get a better handle on that beast, let me suggest this:

Enter the “Lean Startup”

If you haven’t heard about it yet, and regardless of whether you’re actually working at a startup (I’d say that it is even more important for large organization), you need to check out the Lean Startup.

While I won’t be able to do justice to it here, let me use this admittedly gross oversimplification:

You are mistaken about the predicted business value.

Really.

So what you need to do is to start applying the scientific method – a series of experiments in which you are looking for proof to validate your hypothesis about the predicted value, where the outcome of one experiment is used to formulate another hypothesis to be tested in the next experiment.

Let me say this differently – until you are as rigorous in evaluating the predicted value of a given initiative as you are in estimating its cost, with that rigor increasing with the size of the initiative, you have no business starting to work on it.

And this is what’s missing from most of the software development world.

Portfolio management

I don’t really care what you call it, but the portfolio of potential investments and the risk analysis around them needs to be handled properly.

Based on the language I’ve chosen, you can see parallels to the world of finance. Now, before your mind starts going to the news of all the shady crap that’s been going on in the world of finance, understand that there are both positive and negative outliers in every domain.

That being said, I won’t point you to books on finance (at least, not for starters).

I suggest reading Manage your project portfolio, by Johanna Rothman (one of my favorite people in the world). She also has a couple of blogs, and that can help you get started.

This has already gotten quite long, so I’ll skip a bit and talk about what you should do if you’re “just” a developer being asked to give an estimate.

How to give a good fine-grained estimate

Here’s the format to use:

This will take a team of N between T1 and T2 and I am P% confident in that range, with the following assumptions (1, 2, 3), and most importantly, that the team does not work on anything else during that period of time.

Let me repeat that important bit again:

Assuming the team does not work on anything else.

Most of the kinds of people who ask for estimates aren’t going to like that kind of answer. You may get some pushback, “don’t be clever – can this be done by the end of the year, or not?”. In other words, they’re asking you for a commitment – not an estimate. This is common with certain types of dysfunctional organizations – the project management people are only measured on adherence to schedule, not on whether the system solves the right problem. You, the developer, want to make sure you’re solving the right problem – you want to be Agile. But that’s not the point.

Let me repeat – Agile is not the point.

If the organization around you is dysfunctional, be smart – don’t try to be right. Solving the right business problem is the right thing to do, but often it’s not smart thing to do. If you don’t like having to deal with this kind of organizational politics, you had better find yourself a different organization – otherwise, you had better be smart.

How to receive good estimates

Assuming you’re a team lead, project manager, product owner, or something similar, here’s how to respond when someone gives you an estimate in the above format.

1. If P is subjectively high enough and T2 is subjectively low enough, give them the green light.
2. If P isn’t high enough, or the range of T1 to T2 is too broad:
   1. If you have previously given them time to do research (RT), double the value of RT.
   2. Make sure they are only doing research during this time (no other development).
   3. The purpose of the research is to increase P, or decrease the range of T1-T2
   4. When receiving the new estimate, go back to the beginning.

To be clear, “research” does not mean navel-gazing. It can and often will involve writing a bunch of code as well as on figuring out what the requirements should have been in the first place.

“But Udi, won’t this end up wasting time that could be better used on actually building the system?”

No.

In reality, this will end up paying back all the time that should have been spent up-stream on portfolio management and requirements analysis activities.

Which brings me to…

Working software over comprehensive documentation

Sometimes, writing high-quality working software is the last thing you want to do.
Specifically, the earlier you are in the project, the less likely that you should be focusing on code.

Often, the most cost-effective thing to do is some rapid-prototyping like creating UI mock-ups to verify what the system should really do.

This really deserves a blog post of its own (with thanks to Geert for finding the link), but suffice it to say that this is a skill-set all its own that should exist in every software development organization.

On other word on small startups – I often hear from people who are doing their own startup that want to do DDD, CQRS, SOA, and a bunch of other three-letter-acronyms they picked up on various blogs and books, because this time it’s up to them; this time they’re going to do it right.

No, no, no – stop that. Go read the Lean Startup. Then do it. And if you’re extremely lucky, you will be so successful that in a couple of years you will be in the position to rewrite the system – the only difference is that now you’ll REALLY know what the system is supposed to be.

In closing

OK – so that’s well over 1500 words I’ve spilled on the topic.

While the hashtag #NoEstimates makes a great soundbite, estimates are an important part of the information that needs to flow around the organization to help quantify and mitigate risk. Too often there are various organizational dysfunctions tangled up in the same areas as estimates which, I suppose, could give the impression that the estimates are to blame.

While I wish I could tell you that all you need to do is find an organization that doesn’t have any of these dysfunctions and all will be well for you, unfortunately there aren’t any organizations without them. Just like all families are dysfunctional in one way or another, so too are organizations. This is simply because each of us human beings is somewhat dysfunctional.

Well functioning organizations are made up of highly aware individuals – people who have become able to see and mitigate some of their personal issues, and thus can be patient as the people around them similarly work through their issues. Together, these individuals continuously create and adapt their working processes and systems to compensate for the various dysfunctions in the group.

Under these iteratively growing conditions of mutual trust, various kinds of estimates are performed at different levels and times and are just a normal part of communication and decision-making.

Please, don’t throw out the baby with the bath-water.

For everybody who has been building message-driven systems on .NET – I really do believe that this is the dawn of a new era (and if you know anything at all about me, you know that I don’t tend to exaggerate).

What is it?

Imagine designing your solution by graphically creating endpoints and messages, seeing the routing laid out in front of you visually, being able to specify system wide error and audit queues, having your cross-cutting authentication applied automatically to all endpoints, correlating events together into sagas, pressing F5, and as everything springs to life, on your second monitor you can see the messages flowing through your specific debug session synchronized in real time as you step through your code.

Well, you don’t have to imagine – that’s now a reality.

Top that off with some nice production monitoring that includes heartbeats for all of your endpoints, notification of message processing failures (with the ability to send them back for reprocessing), and some cool extensibility to plug in your own custom checks.

It’s a whole new world of service-oriented development.

This short video will give you a sense of what it’s like:

Here’s some other things you might find interesting.

Testing you can trust

I’ll bet you didn’t know the level of testing that we’re now putting into each release.

Every time we put out a new release, we test it on cloud servers configured across all permutations of transports, containers, persistence technologies, server operating systems, as well as for compatibility with previously released versions 2 years back (!). This way, when you want to have a new system subscribe to events published by a system already in production, you’ll know that it will all “just work” – no matter what.

Running all of these tests in the cloud is something that we could never afford back when we were a regular free/open-source project, but is definitely worth it. We know you count on us for your mission-critical systems and will continue doing our very best to be worthy of that trust.

The roadmap

Now that the platform is out, the next big thing you can expect is the removal of all distributed transactions.

Over the past months (and even years) we’ve been making preparations for this, making all sorts of small backwards compatible changes and enhancements so that we can fully control the inbound and outbound behavior of messages. The investments we’ve been making in automated tests is also in support of this. We want to be absolutely positive that you will get the exact same logical behavior as before without losing any of the transactional guarantees you’ve come to expect from NServiceBus.

We’ll give you updates on this and the other things we’re working on soon, as part of our larger roadmap so that you’ll know roughly what kind of big features you can expect and when. Of course, we always leave some “slack” in our iterations for working on critical patches, enhancements, and refactoring.

In any case, the main idea here is for you to be able to give us feedback and input on what’s important to you and for us to communicate back what we will be able to commit to and when.

Expressing our thanks

It’s been quite a journey this past year – the company has doubled in size and we’ve really got a great group of people here. While we’re constantly looking ahead and seeing all the wonderful things that we want to do, I wanted to take a moment and look back.

It’s been almost two years of work on ServiceMatrix (previously named NSB Studio) – the absolute biggest investment we’ve made. This past year was really the biggest push, and when taken together with all of the other tools – the production monitoring in ServicePulse that so many people have been clamoring for; the under-the-hood visibility of ServiceInsight; and the unseen glue of ServiceControl tying them all together; well, it was a very ambitious year.

I wanted to thank everyone who took part in our betas – your feedback was absolutely invaluable;

To our Community Champs who keep asking us the difficult questions and pushing us to do better;

To all of you in our community who have believed in us;

To the growing ranks of paying customers who voted with their wallets as well as their hearts;

You have my personal thanks as well as those of the entire team.
It continues to be an honor and a privilege.

Go and get it

That’s it – go to www.particular.net and click the big “download now” button and take the new platform for a spin!

I can’t wait to hear what you all think.

I’m happy to say that the recording of that session is now online, courtesy of the good folks at Skills Matter, and you can now view that here.

In this presentation, you’ll also see a lot of the new capabilities that will be coming out soon with the release of our new Particular Service Platform – the suite of tools that makes NServiceBus that much more awesome. From monitoring of production issues, to debugging cross-endpoint message flows, to modeling of message-driven & service-oriented systems, it really takes distributed systems development to the next level.

Check it out!

Oh, I almost forgot, there are still some seats left for my course in New York City. Details here.

I’ve given my Programming in 4D talk already several times and I haven’t yet uploaded the code for it.

And seeing as I’m going to be giving it again today (at the DevWeek conference in London), I figured that I should finally get my act together and put it online.

Interestingly enough, the other conferences I’ve spoken at either didn’t record it or didn’t put the recording online. Hopefully DevWeek will do better <FingersCrossed/>.

The overall solution

The scenario I talk about in this presentation is (again) a standard one in the world of retail: customers who want to return products that they’ve purchased and get a refund.

I’ve used our new ServiceMatrix tooling to model the solution like this (click for a larger image):

If you’d like to download the complete solution, click here.

Now, we’re going to focus on the RefundPolicy object that you can see towards the bottom left.

The Refund Policy

So, in this scenario, what we’re going to implement is a process whereby if you return your products within 30 days of the purchase, you’ll receive a 100% refund; if you return your products within 60 days you’ll receive a 50% refund, and anything longer than that and no refund for you.

Here’s the code:

public class RefundPolicy : Saga<RefundPolicySagaData>, 

                            IAmStartedByMessages<OrderAccepted>, 

                            IHandleMessages<ProductsReturned>, 

                            IHandleTimeouts<Percent>, 

                            IHandleSagaNotFound

{

    public void Handle(OrderAccepted message)

    {

        Data.OrderId = message.OrderId;

        Console.WriteLine("OrderAccepted");

        Data.Percent = 100;

        RequestTimeout(TimeSpan.FromDays(30), 50.Percent());

        RequestTimeout(TimeSpan.FromDays(60), 0.Percent());

    }

    public void Handle(ProductsReturned message)

    {

        Console.WriteLine("ProductsReturned");

        Bus.Send<IssueRefund>(m => m.Percent = Data.Percent);

        MarkAsComplete();

    }

    public override void ConfigureHowToFindSaga()

    {

        ConfigureMapping<ProductsReturned>(m => m.OrderId).ToSaga(s => s.OrderId);

    }

    public void Timeout(Percent state)

    {

        Console.WriteLine("Timeout");

        Data.Percent = state;

        if (state == 0)

            MarkAsComplete();

    }

    public void Handle(object message)

    {

        if (message is ProductsReturned)

            Console.WriteLine("No refund for you");

    }

}

public class RefundPolicySagaData : ContainSagaData

{

    [Unique]

    public int OrderId { get; set; }

    public Percent Percent { get; set; }

}

And what’s so good about that?

Well, not to steal my own thunder and give you a reason not to watch the video when it comes out, the trick is in the two RequestTimeout calls that are invoked when an OrderAccepted message arrives. You see, what that does is that it takes the data that defines the behavior of the refund policy and persists that via a queue, which will play back the data to our object according to the defined schedule.

This way, if/when the business decides to change the rules (say, reducing the timeframes to 20 days and 40 days), that will only affect users who are placing new purchases. The customers who made a purchase 50 days earlier (when the rules were still 30/60) will get the correct behavior applied to them.

Next steps

Of course, nobody would want a developer to have to open this code and change it in order to make a simple change like 30/60 -> 20/40, so we could externalize the data by creating a dictionary property on the saga where the key is the TimeSpan and the value is the Percent. Then, we could pull those values from either a config file or database and inject them into the property on the saga.

Here’s how the code of the saga would change:

public class RefundPolicy : Saga<RefundPolicySagaData>, 

                            IAmStartedByMessages<OrderAccepted>, 

                            IHandleMessages<ProductsReturned>, 

                            IHandleTimeouts<Percent>, 

                            IHandleSagaNotFound

{

    public Dictionary<TimeSpan, Percent> DataDefinitions { get; set; }

    partial void HandleImplementation(OrderAccepted message)

    {

        Data.OrderId = message.OrderId;

        Console.WriteLine("OrderAccepted");

        Data.Percent = 100;

        foreach(var kv in DataDefinitions)

            RequestTimeout(kv.Key, kv.Value);

    }

And the code to do the property injection would look like this:

public class RefundPolicyDataConfigurator : INeedInitialization

{

    public void Init()

    {

        Dictionary<TimeSpan, Percent> data = null /* get from config instead */;

        Configure.Instance.Configurer

            .ConfigureProperty<RefundPolicy>(p => p.DataDefinitions, data);

    }

}

Classes the implement INeedInitialization are invoked at process startup, and that call to ConfigureProperty instructs the container to set the DataDefinitions property of our RefundPolicy every time it is resolved (which is on every message).

In closing

We now have a solution which allows non-developers to make changes to the refund policy definitions without requiring us to deploy any new code to production. Also, any changes that are made will preserve the promises we made to our users in the past.

Of course, if you want changes to rules to impact all users immediately, this approach wouldn’t be so good.

Again, if you’d like to download the complete solution, the code is here, and if it wasn’t already obvious, this code makes use of NServiceBus quite heavily.

When the recording comes online, I’ll update this post with a link as well as do another blog post.

Hope you like it.

But since I’ve been getting repeated requests to weigh in on the topic, I guess I’ll have to risk fanning the flames up again.

Also, since quite a few reactions have already been written on the topic (and I don’t want to repeat them here), I’ll just point to this post by Arnon which sums them all up pretty well.

Now, I don’t entirely agree with all the commentary Arnon pointed to, or all of his thoughts on the topic, but I’ll try to take those up some other time.

And before jumping into it, let me say that there is a lot of good stuff in the article and that, regardless of naming, spreading the word more broadly on these approaches has value.

So, where do I stand on the topic

First of all, for those of you who have been following my blog for a while I’d say this:

Microservices almost equals Autonomous Components.

Why “almost”?

Because an Autonomous Component (AC) isn’t necessarily a physical unit of deployment – very often we’ll see multiple ACs deployed in the same physical process. One of the most common occurrences is in a web front end built as a composite UI. In the same web server process we’ll see components from multiple Services.

This is something that was hardly mentioned in the original article.

On Services and Systems

In my world, Services are a larger organizing principle that are meant to align solution domain boundaries with problem domain boundaries.

Now, that might sound very similar to this passage from the original article:

“The microservice approach to division is different, splitting up into services organized around business capability. Such services take a broad-stack implementation of software for that business area, including user-interface, persistant storage, and any external collaborations. Consequently the teams are cross-functional, including the full range of skills required for the development: user-experience, database, and project management.”

Now, this isn’t entirely surprising because I did have several conversations with both James and Martin on the topic over the past couple of years.

Still, there is something important missing here that I believe is very important to achieve loose-coupling, and that is that Services necessarily have to span system boundaries.

Let me repeat that: a Service will need to have components that are deployed to more than one system.

Here’s why:

Let’s say you have a piece of data like the price of a product. Not only will that data be visible in one system, often it will need to be shown (as well as updated) in other systems too. In order to have appropriate encapsulation of that concept, the owning service will need to be the one that owns the components that operate on that concept in the other systems.

This means that if we need to show the product price on an invoice in a back-end system, then that invoice would have to be a composite UI as well, and the service which owns the price will have a component deployed there which would be responsible for showing the price on the invoice.

In this manner, no code outside the service boundary would know about the concept of the product price and thus could not end up coupled to it.

Although the original article does get into this to some degree (when talking about Decentralized Data Management), I don’t really see how a microservice/AC could end up having this level of ownership of data.

Still, the point made about different persistence technologies is valid at the level of services (though not ACs).

How big is a service

Now, if the price is not shared outside the boundary of the service, then how would order totals be calculated?

The answer is that the totals must (MUST) be calculated in the same service.

This shouldn’t be surprising as it’s just good old OO – encapsulating data with the logic that operates on it. Or, if you’d like, call it the Single Responsibility Principle: there should only be a single service impacted by a change to the definition of this data.

As a result, you’ll tend to see services that aren’t all that small, and probably not so many of them. In my experience, I’ve seen between 7 and 15 services the majority of the time.

Cross-service collaboration

Although I am glad to see the recommendation for event-based interactions between microservices, the focus on cross-process communication ignores some extremely important collaboration scenarios – the most important of which is in the client tier.

In a web application, it is quite common to have components written in javascript from multiple services interacting among themselves in the browser – one publishing events, others subscribing to those JS events. It is also quite common to see those JS components request some data from back-end components in the same service in response to those JS events.

This type of synchronous RPC communication within a service boundary is perfectly acceptable, although it stands in contrast to the recommendations of the microservices approach.

Caveat on sharing data

I’ve been going on and on about the importance of not sharing data, but there is one exception to that rule.

There is a special service that I call IT/Ops which (among other things) is responsible for integration with 3rd party systems. As a part of this integration, it encapsulates data transformation logic and, as a result, needs to be able to receive data from the other more business-centric services.

As you can imagine, this puts IT/Ops in the risky position of coupling itself to a lot of things and thus needs to be done carefully. As a result, I recommend that many of the most skilled technical people work within the IT/Ops team, also serving in a consultative capacity to the other service teams.

In closing

I am extremely thankful to Martin and James for writing the Microservices article.

I think that the conversations it has sparked are timely, and hopefully more people will ponder these questions of how to structure their code-bases in order to avoid them becoming monolithic.

And while I think that it’s great to consider aligning team boundaries with service boundaries, people need to understand that it will need to be an evolutionary process – it will take time to transition an existing code base and an existing team structure to this new model, especially since these teams will have to continue to deliver features and bug fixes through the transition period. Jumping to the new model directly may cause more harm than good.

This is actually one of the most salient topics of my course (next one in NYC in May) – how do you get there from here. In my experience there are 4 phases that companies go through, often taking at least a couple of years, with larger environments taking potentially longer.

In any case, let’s keep the conversation going.

What are your thoughts? Have you been applying the Microservices approach, or possibly the one I talk about (I really should give it a name). What’s been working well for you? What hasn’t?

Leave me a comment or write your own blog post.

Well, it would be more accurate to say the first ADSD UNconference.

For the thousand+ people I’ve trained over the past 7 years who have been applying the principles of the course and want to discuss what’s been working for them, what hasn’t been working, and how new tools and technologies can be applied to achieve loosely-coupled bliss, this is the place for you.

Important: This is unconference is limited to people who have taken the red pill ADSD course.

Join us in London on June 25 – register here.
Or in New York City on October 1 – register here.

We will be meeting the evening before to set the agenda and then go out for drinks and dinner.

What’s ADSD?

That’s the acronym for the Advanced Distributed Systems Design with SOA and DDD course I’ve been teaching for the past 7+ years.

If you want to know about Service-Oriented Architecture (or Microservices as it’s now being called – separate blog post coming on that topic), Domain-Driven Design, Command-Query Responsibility Separation, and how to gradually transition your way there – this is the course for you.

The next one is in London March 24-28 – register here.
The one after that is in NYC May 12-16 – register here.

What’s an “unconference”?

Taken from Wikipedia:

An unconference is a participant-driven meeting. The term “unconference” has been applied, or self-applied, to a wide range of gatherings that try to avoid one or more aspects of a conventional conference, such as high fees, sponsored presentations, and top-down organization.

Typically at an unconference, the agenda is created by the attendees at the beginning of the meeting. Anyone who wants to initiate a discussion on a topic can claim a time and a space. Unconferences typically feature open discussions rather than having a single speaker at the front of the room giving a talk, although any format is permitted.

In closing

I know some people have gotten so much value from the course that they’ve actually attended the course a second time.

Most probably wouldn’t be able to convince their boss.

So it is my hope that this format can serve as a kind of continuation to the course for people in the ADSD community to maintain contact and learn from each other, on top of the online discussion group.

Hope to see you there!

<Applause/>

It’s been a long time since my first blog post on NServiceBus (about 7 years ago) and the community has come a long way since then.

In any case, this won’t be a “looking back” post but rather one looking forwards.

In addition to myself and the rest of the crew, we’ve got some great speakers coming out including the inimitable Greg Young and the formidable Ayende Rahien (aka Oren Eini).

Register here.

For those of you who can’t make it to London for June 26-27, we’ll be running another NSBcon in The Big Apple (New York City) on September 29-30. Register here

If you’d really like to come but have to pay out of pocket (either because you’re a contractor or your employer just won’t pay), send an email to nsbcon@particular.net and we’ll get you a nice discount, but please hurry as we don’t have many of these tickets left.

Hope to see you there!

I’m afraid that the venue we’ve got only has 24 seats and about half of those have already been sold, so if you’d like to register, please don’t wait.

A while back, an attendee named Tom Cabanski blogged his experiences going through the course – it’s a really interesting view into the roller-coaster of the course. He’s broken it down into several parts, starting here, but I’d say that part 4 is probably the most interesting.

Here are some choice quotes:

Don’t name your services. You probably will get the name wrong at first and that will lead you to group the wrong things into the service because the name will insist you do.

By day 4, every time Udi mentioned “best practice” there would be a ripple of giggles in the room.

Hope to see you there.

Register here

I’ll be teaching the 5-day Advanced Distributed Systems Design course on the week of March 24-28 in London. Registration is currently open here.

I’ve also decided to come back to New York City and teach the course there on the week of May 12-16. Registration isn’t open just yet but if you’d like to reserve a spot, send me an email to training@udidahan.com.

You might have also heard some whispering of an NServiceBus conference – that too will be coming this year.

Stay tuned.

While we’ve been making some pretty amazing progress, it still a bit premature for big announcements.

In any case, here’s a partial list of goings-on:

Monitoring – sneak peek

One of the most requested additions to NServiceBus was strong production monitoring capabilities.

Some of the things we have coming include:

1. Endpoints periodically send heartbeats to a central location, and if they stop coming, the administrator gets notified.
2. If messages can’t be processed for some reason and are sent to the error queue, the administrator gets a nice visual notification with the ability to send the message to be reprocessed.
3. Developers can also plug in their own custom code to periodically report on various things, for example the existence of a folder, accessibility of a URL, etc.

Check out this short video to see how it all works:

And yes – it will work across all our supported transports (MSMQ, RabbitMQ, ActiveMQ, SQL Server) as well as in the cloud (on Azure).

Partners

This was also a big year for the larger ecosystem.

In addition to our long-time partner Headspring, home of the legendary Jimmy Bogard, many more companies are now Official Partners.

I don’t want to play favorites here so this list is not in any particular order (except alphabetical). There are also a number of independent consultants that do amazing work but I’m not including them here for brevity so you might see some prominent countries not listed.

In the US, we have:

• Architecting Innovation which covers all of the US.
• CTS – focusing on the South East.
• Headspring – Nation-wide but with a Texas flavor.
• ILM – focusing on the Mid-West.

ILM also happens to be the home of David Boike – one of our community champions and author of the first NServiceBus book.

In Europe, we have:

• BBV – focusing on Switzerland, Germany, and Austria
• InfoSupport – focusing on the Netherlands
• WebStep – focusing on Scandinavia

BBV were the ones who did the heavy lifting for getting our support for ActiveMQ off the ground.

And not to forget the folks down under, there’s Driven Logic in the Sydney area.

Keep an eye out as there’ll be some interesting things coming from our partners this coming year.

In Closing

I’ll close with some thank-you’s:

Thank you to the hundreds of new paying customers, as well as to the customers who’ve stuck by us over the years who make it possible for us to make enterprise development on .net that much more enjoyable.

Thank you to all the employees at Particular Software – now twice as many as there were last year, and probably going to double again in 2014. This team, I’ve got to tell you, is absolutely world-class.

Thank you, my dear reader, for coming back and reading my rambling thoughts time after time. I can’t believe I’ve been doing this 10 years now.

And, to my wife and kids, well, they don’t read my blog, but on the off chance they might one day, I’d better thank them as well.

I wish you all a wonderful 2014.