Jak Charlton - Insane World

We Are Not Doing DDD – Part Two - CQS

Jak Charlton — Mon, 22 Jun 2009 16:32:58 GMT

From the beginning of my current project we have been working under some horrible constraints, many imposed by legacy systems, many by late decisions that would have speeded things immensely if made earlier, and many imposed by decisions that are outside of my control.

This lead us early on to make decisions on architecture that eliminated any possibility of using DDD heavily, and as I mentioned in my original post about this project it is also essentially a glorified CRUD system - we read data from databases, we modify it a bit, and we put it back again.

What We Are Doing – Command Query Separation

The first architectural choice that I made, and in hindsight got accepted with relative ease, was to employ CQS to simplify the system.

Earlier projects had suffered quite badly from being abstractions of legacy systems with new functionality bolted on. As

far as possible I wanted to eliminate this problem from our project, as I was all too aware that this was an easy trap for the team to fall into, and would burn a lot of development time very fast.

To make the system simpler I chose to separate our query mechanisms from the "domain" type stuff, our commands and data writing/updating code.

CQS is a pretty simple concept. One path through your system is responsible for querying of anything much more than "Get By ID" type calls. This side of our system is responsible for reading data from legacy systems, reading data from legacy web services, and for reading data from our application database.

Querying

The Query side deals largely in DataTables, DataSets and a very very limited number of DTOs. This data is largely and essentially used for display purposes, it is the contents of search results, the results of postcode lookups, the information to populate dropdown lists and tables.

The Query side of CQS does not need strongly typed entities, nor does it require strongly typed DTOs - as it is largely ad-hoc data maintaining these entities and DTOs would consume a disproportionate amount of development time for something a DataTable can deal with more than adequately.

Our Query side actually uses the Query Object pattern, most of which encapsulate a call to an Oracle DB, some of which wrap simple NHibernate criteria and a few of which sit around web services. These are fronted by a WCF facade, which allows us to put these queries out of process, and to cache them aggressively should be need.

What we do not have in the system anywhere, and especially in the Query side, is any big entity model – we have two very small entity models (5 entities in total) used for very specific functions within our application.

Command

The Command side of the CQS equation is a little different. Firstly, it deals with anything that is essentially a request to "go do something". In our case these are commands like "SaveDraft" and "CancelPolicy"

The Command side can write data to the databases, the Query side is not allowed to.

If we were doing DDD, this is where our Domain would sit - and in fact we have been referring to this as "the domain" for most of this project. But, we aren’t using DDD and we don't really have a Domain Model. It is however where most stuff we could describe as "logic" lives. This is primarily a bunch of application services, most of which operate in response to messages fired over MSMQ using NServiceBus.

This side of the architecture has a number of listeners that are responsible for pulling messages from various MSMQs and for dealing with the specific requests, for example pushing data back to our legacy systems, for logging, and eventually for notifications.

Benefits

Early on, the use of CQS paid for itself easily. By removing the complications of maintaining read and write models together, we quickly managed to get working prototype up and running, with no need for a "domain" or entity model up front.

As the project has progressed, this decision has helped simplify decisions that otherwise would have involved structural or fundamental changes to the architecture – but more importantly, it has helped all of the developers think in terms of a command driven system, rather than viewing the system as CRUD over a database.

Commercial Suicide - Integration at the Service Level

Jak Charlton — Tue, 19 May 2009 07:03:00 GMT

In my previous post I described the problems with trying to integrate your organisation at the database level, and the fallacies surrounding the idea of the One True Authority Database. I also alluded to this being a problem with services too.

When you try to create a monolithic and authoritarian database that knows all, sees all, and brings everything into a single coherrent whole, you are heading down a slippery slope. Different applications have different requirements, and even within the same job role, the meaning of Customer can have a totally different meaning in two different contexts. I also argued that data has no value without context, and the context is the application that is meeting the business requirements, be these user interaction, system interaction, or reporting.

SOA to the Rescue!

I'm not arguing without ample evidence behind me here - the idea of integration at a database level was one that was well discredited many years ago. I grant you many organisations haven't got the message yet, and that many vested interests still keep trying to make the magic solution work, but fundamentally a long time back we in IT recognised that the One True Authority Database (OTADB) was a "bad thing" - and this lead us to the promised land - initially Web Services and then on to Service Oriented Architectures.

SOA is an attempt to resolve the OTADB issue - take all those database calls, and even calls to different databases, and unify them through a set of coherrent and cohesive services - SOA can provide us with the abstraction over out data stores that is required - and now we can have the One True Authority Service Layer.

With SOA we can define services to wrap our databases, services to update data, service to read and query data, and even services to unify those "different" Customers into a single coherrent and canonical data model.

And that's where it all goes wrong again.

Anti-Pattern: The Canonical Data Model

Back when our database guys were trying to integrate all this information, what they were trying to achieve, but maybe didn't realise, was a canonical data model - this is the "holy grail" of SOA. Martin Fowler describes it thus:

How can you minimize dependencies when integrating applications that use different data formats? …Design a Canonical Data Model that is independent from any specific application. Require each application to produce and consume messages in this common format

Sounds perfect! We now have a way of abstracting our systems from each other, and our canonical data model will pull this information all together at the end.

So, I'm going out on a limb here, disagreeing with Martin Fowler is rarely a smart move.

Except I'm not disagreeing with Martin Fowler so much as I am disagreeing with those that try to implement his ideas and principles. I don't recollect anywhere that Martin says that all data can be squeezed into this model, or that this model is the "one true view". In fact, later evolutions of Martin's work have included things like Domain Driven Design, which is almost the opposite of a canoncial data model - DDD says that you have a different model for every domain or context, each specifically engineered to solve the problem at hand. Given the choice, I am certainly in the DDD camp here - after all data without context is worthless, and it is the domain (and DDD) that provides the context.

The key and important part of a canonical data model is it provides a mapping between inconsistent formats and differing views of data.

So as an ideal, the canonical data model is fine - in practice it is often a major anti-pattern.

There is still no One True Authority, and SOA really never promised this, nor did the canonical data model - what they promised was enough abstraction to allow applcaitions to work independently, and to fulfill their specific requirements independently. And then to leave it to the service layers to unify this data.

So Service Oriented Architectures HAVE Come to the Rescue?

Not quite - again my argument here isn't so much about SOA, or any of the things around it, but more the people who try to implement it, often without understanding what they are trying to achieve, the problems that SOA or a service layer solves, or even the amount of effort that is required to get a canonical data model right.

The key part here is that a canonical data model is a massive excercise to do right. Essentially it means analysing every applciation, every service and every database within your organisation and extracting their individual requirements and current functionality, and trying to distill that down to a set of common definitions.

While this may be eventually achievable, in practice it is almost never the case. This "holy grail" of SOA is much like the "holy grail" of integration at the database level, eventually it boils down to "the problem is too complex, too fast moving and too indeterminate to define". In development we figured out long ago that we needed Agile to help us move quickly as requirements shifted and changed, but the OTADB and the SOA canonical data model do not allow this to happen, they seek to solidify the design early.

SOA done "right" may well make this problem smaller, but SOA is rarely done right.

Can Domain Driven Design Help?

DDD was largely an attempt to address these problems - by identifying that individual applications and contexts required different data, DDD allowed us to recognise that square pegs and round holes do not match. Eventually with a big enough hammer you may be able to force it through, but there is going to be a lot of effort and a lot of collaterel damage.

Domain Driven Design includes terms like "Domain", "Bounded Context" and "Anti Corruption Layer" specifically to deal with the issues around trying to use a "one solution fits all" hammer to crack what is quite a simple nut.

DDD is largely unconcerned with what other things Domains talk to, or even how, all that matters is that within a Domain there is one logical view of the world, tailored to the requirements of that specific Domain. Beyond that Domain, there may be a large central database, there may be a perfect canonical data model, but at application level, we have the Domain, Bounded Contexts and Anot Corruption Layers to protect us from all of that.

DDD gives us a loosely coupled way of thinking.

So SOA is Evil and the One True Authority Database is Evil?

Well hardly, but unfortunately we live in an imperfect world, and the people implementing these noble ideas are often far from perfect. They do not have mystical powers of future vision, and they do make mistakes.

By creating properly decoupled systems, and in this I certainly do not mean "systems that use web services", we can isolate our applications from external change and influence, and ensure we meet the requirements we have today, and can react quickly to future requirements as they appear.

Big Design Up Front is still BDUF, whether it is done at a database level, a service level, or in a canonical data model.

Commercial Suicide - Integration at the Database Level

Jak Charlton — Thu, 14 May 2009 06:44:00 GMT

There are many ways you can commit commercial suicide, but there is possibly no slower and more agonising death than that produced by attempting that great architectural objective, the single authoritative database to which all applications talk.

The theory is good, if we have a single database then we have all our business information in one place, accessible to all, easy to report against, reduced maintenance costs, consistency across all applications, and a host of other good objectives.

However all these noble ideals hide a more fundamental problem, that the single database does not solve any of them, and makes most of them into far bigger problems.

All Information In One Place - The Single Authority

On the face of it this sounds like a great objective - after all developers try to live by the maxim of Don't Repeat Yourself - and data in many place is a clear violation of that principle.

Data that appears across many applications and across many storage mechanisms leads to all sorts of massive problems; inconsistency, duplication, replication, duplicated business logic and code, essentially all boiling down to - you end up with spaghetti data. Spaghetti data is much like spaghetti code - it sprawls, gets tangled up, and becomes hard to pull apart without covering yourself in pasta sauce. This is obviously a "bad thing".

So what is wrong?

Well the first and most obvious thing that is wrong is that all applications have different requirements, and different "world views". Although there may in theory be some concept of a "Customer" for the organisation as a whole, even this most basic of data items varies widely between individual departments and even individual applications within a single department.

You could approach this problem, as many database centric people would do, and say "that is the problem right there, we need to standardise all these applications to use the One True Customer". But that is missing the really important word in the definition of the problem ... "requirements" ... this is not accidental that the Customer is different to different parts of the business, it really *is* different.

Your database guys could say "well your Customer in your application may be different as you have different requirements, but you will just have to fit it into our One True Customer", but this is then like trying to put snakes into a plastic bag - they really don't want to be in the bag, they don't fit too well in the bag, and sooner or later while putting one in some others are going to escape or bite your hand. And when your other department starts putting his snakes in the bag too you will be fighting for who gets to not be bitten.

And worse still - now you are trying to map from your requirements based Customer to the One True Customer, and spend an inordinate amount of time maintaining this translation layer in your application. When that One True Customer changes, as he undoubtedly will as new applications require he is expanded to deal with more data they require, every previous application needs to be revisited, large parts of it need ot be re-written, and the whole application needs to be regression tested again. And you have to do this for every single application talking to your single authority database.

You could just skip this stuff, and rely on your applications ignoring this new data, and rely on the database not caring if they correctly updated new data - but this really will come back to bite you - when that bag of snakes starts getting really large and really full - you really don't want to be the one trying to get new snakes into it.

The Truth About the One True Authority

There isn't one.

There - I've said it - I have upset all those database guys, probably upset a large number of SOA guys (I'll cover Commercial Suicide - Integration at Service Level in a later post), and have totally disagreed with noble business objectives.

The truth is, data must have context - without context, data is worthless, absolutely and totally worthless. Data stored in a database has no context, and therefore has no value. Context is provided by the applications that read and write that data, and therefore they are the only thing that matters, and their requirements are the only thing that matters. That means, they need data that is specific to their application, structured in a way that makes that application meet the business objectives, and in a way that makes that application meet non-functional requirements like resilience, reliability and consistency.

So Why Does Anyone Want the One True Authority Database?

Well, in legacy terms it is easy to understand why database admins and database developers want it - it is their lifeblood, their whole raison d'etre. More importantly, it is the culture in which they were brought up - the data is the important thing, the data is the centre of the universe, the data must be consistent, uniform and pure.

But leaving database developers aside, more importantly why would a corporation want the One True Authority Database (OTADB)? After all, the title of this post says this is "Commercial Suicide", so why hasn't this got through to management?

Well - the promise of OTADB is that it will reduce errors in duplication, reduce waste, reduce duplicated effort and reduce maintenance costs - all highly desirable business objectives. And indeed, from those that advocate this approach (those database admins and developers again), the OTADB sounds mighty attractive. On the face of it, it achieves all of these objectives.

Where it falls down is that this holy grail of software development is always just out of reach, they never quite manage to achieve it. Each application that is developed starts to make the OTADB worse, people start to hack things into it to get it to meet business requirements, not because developers want to hack things together, but precisely becasue the restriction of the OTADB *forces* them to do it that way if they are to deliver any kind of functionality at all.

They blame these hacks for ruining their vision of the One True Authority Database, the database admins tell them that they have to fight the application teams to stop them messing up their nice database, but that the OTADB no longer meets the noble objectives as those pesky development teams have messed it up for everyone.

Wait a Minute - What is the BUSINESS Objective Behind the One True Authority Database?

If anyone was to step back from those noble objectives and ask a far more fundamental question, the solution might actually be a lot more obvious than it may seem. While they are all noble objectives, largely actually made worse specifically by the OTADB approach, they are not the real business objective.

Underlying all the other requirements, the ultimate business requirement that drives people (in particular database admins) to want a single database is so that they can see what their company looks like, in other words - so they can produce Management Information - reports to you and me.

This is the single and most fundamental requirement for a business - to have a clear, consistent, accurate and up to date picture of what their company looks. This is what management needs, it is what allows them to make decisions, allows them to identify problems and allows them to spot opportunities.

So, we are going to all this effort, and believe me it is extensive and significant effort, all to support some reporting tools at the end of the day. Reporting tools have problems with data in different formats, with data that is inconsistent, with data that is disparate and distributed. So at some point in the past, the "accepted truth" became "we need one true authority database to be able to produce good reports"

Reporting is a Context - and data only has purpose and relevance in context

If the Elephant in the Room is Actually Reporting, How Do We Solve The Elephant Problem?

This is almost so easy to deal with, it is silly. Perhaps it is because it is so obviously simple that is has been overlooked by many and rejected by others. Especially as it violates another one of those noble objectives ... to provide quality reporting information, we duplicate more of our data.

Yep - we duplicate it - after all reporting data is read only, so it doesn't matter if it is just a copy of other data. Reporting requirements are also very different to transactional requirements too, so we get the added benefit of being able to optimise that duplicated data for the reporting functions.

Data in relational databases is actually very poor for query and reporting purposes, and there is a constant compromise to make it fast for all those applications to write to, that makes it poor to report on - and vice-versa.

How this data gets into the reporting database isn't my direct concern in this blog post, suffice to say the "easy" way is to publish messages with data changes, and have a reporting application pick those up and persist them. My point here - is that splitting the reporting functions from the day-to-day business functions pays massive dividends.

Now We Have a New Problem

That still leaves us with one problem - what happens when disparate applications really do need to know about data in other applications? What happens when my call centre operatives are asked to update the address for one of the Customers. Now as each application has it's own view of the world, and it's own data stores, my accounting applciation does not have access to that change.

Well, the solution to the "how does data get in the reporting databases" question is exactly the same one here - you published messages from your application when you have changes that the rest of the corporation may be interested in. Fire off a message saying "CustomerAddressUpdated" and any other applciation that is concerned can now listen for that message and deal with it as it sees fit.

As It Sees Fit

And this is the real business objective we were trying to achieve in the beginning ... avoiding Corporate Suicide.

When applications are each responsible for their own data, their own actions, and are only responsible for letting the "enterprise" know they have made some changes that other things may need to know about - then you have your solution.

In good development terms, we have proper separation of concerns ... applications are responsible for their data, and their data only. They decide if they care about data from other applications - they are not forced to use it, nor to work around it.

Twitter Went And Stuffed Replies!

Jak Charlton — Wed, 13 May 2009 06:48:00 GMT

I'm not going to spend too much time describing the problem ... Kevin Pang has done so admirably here, however I do want to mention this "small" change as it directly affects a new site I am building right now to work over Twitter.

Twitter's latest blog post titled Small Settings Update states the following:

We've updated the Notices section of Settings to better reflect how folks are using Twitter regarding replies. Based on usage patterns and feedback, we've learned most people want to see when someone they follow replies to another person they follow-it's a good way to stay in the loop. However, receiving one-sided fragments via replies sent to folks you don't follow in your timeline is undesirable. Today's update removes this undesirable and confusing option.

What this essentially means is that the "viral" nature of Twitter has been largely disabled. No longer will you see interesting replies from people you follow to complete strangers and think "maybe that guy is worth following too, I'll check him out". If the person is only mentioned, then you will still see the message, bbut I would lay a large bet this is only a very small percentage of the Replies.

I have no idea if Twitter did this to save bandwidth, or for some other more obscure reason, but ultimately it will hurt Twitter by slowing growth, as growth is largely by "discovering" people from replies. I'm sure "new" applciations will appear to try and plug this gap, but it is so deep in the Twitter API that it cannot be countered easily by other tools.

From my own perspective, it seriously damages the growth potential of my new Twitter based site, so will leave us with other less palatable options like spamming people or creating follow bots to try and get people interested - neither of which is desirable or in my ethos.

Dear Twitter, change it back please while it's not too late!

Dear Twitter users, please express your feelings via #fixreplies ... and remember you can follow my mini-rants @JakCharlton if this one wasn't enough for you!

I Got Bored Yesterday

Jak Charlton — Fri, 08 May 2009 07:46:00 GMT

Maybe something to do with lack of sleep, but I decided to use my middle name, Jak, from now on instead of Casey. So if you wonder why on earth you are following somebody you don't recognise ... it could be me.

TweetSharp Has a New Demo Application

Jak Charlton — Wed, 29 Apr 2009 09:19:00 GMT

If you don't know about the TweetSharp library (google code) started off by Daniel Crenna and Jason Diller then, if you are interested in talking to the Twitter API from C# you should check it.

Yesterday I contributed a small MVC sample application to the project - I have been playing with a small "business" idea and it involves Twitter interaction. As a playpen I started off an MVC application, and then it started growing. It also happens that it has proved to be the playpen for a lot of the MVC/JSON/JQuery interations that my paying job is using right now, so it got even more legs due to that.

The sample application actually shows little about ASP.NET MVC, as almost all of the "heavy" work is done via JQuery Ajax calls, and the core View Engine stuff in MVC only provides the shell pages. It does however show quite a nice way (IMHO) of writing "Web 2.0" applications using ASP.NET MVC.

So if you are interested, go check it out ... all comments welcome, and I'm sure Daniel would appreciate anyone else who would like to commit to the project, either by making the sample applciation better, or by helping write, document and test the TweetSharp API itself.

DDD: Videos from Øredev

Jak Charlton — Sun, 26 Apr 2009 18:36:14 GMT

Leonard Petrica kindly pointed out that the videos from Øredev might be useful to those interested in Domain-Driven Design. They include talks from Jimmy Nilsson, Randy Stafford, Dan Berg Johnsson, Einar Landre and Eric Evans.

So if you are interested, check them out

We Are Not Doing DDD

Jak Charlton — Wed, 08 Apr 2009 20:34:00 GMT

Recently I joined a new project, and within reason, and excluding some legacy systems we have to talk to, we have the “luxury” of an almost greenfield project. Probably as greenfield as you realistically get anyway.

I was brought in partially as a good old fashioned coder (the project needed another person cutting code) and partly as guys in the team knew my blog and general approach to software development and wanted to inject some of those ideas into the team. I’m not sure exactly what they were expecting, and I’m not sure if they got what they were expecting, but suffice to say the first two weeks has been turbulent and frenzied.

I’m certainly not known for my subtlety, nor for my coding skills which are by no means the best you can buy in. What I feel is my key strength is my ability to look at the wider picture than sometimes teams have the luxury to do, and to push some of my enthusiasm and energy into a project. I’m sure I’ll blog in the near future about the actual impact my presence has had on the team and the project, but this blog is primarily about why we are not going to be doing DDD.

But Casey Does Domain-Driven Design

One of the factors that made the client hire me was my knowledge around DDD, or at least my projection of that knowledge on the web. It was certainly something they felt could have helped some of their past and ongoing projects, and something they thought would be a real help to them. I don’t think anyone suspected I would come in and say “Hey guys, lets all do full on DDD”, so luckily for them I didn’t, what may have surprised them is that I did pretty much the opposite. In fact my early assessment was this project really didn’t have any need for most of the stuff in DDD – fundamentally it, like the other projects in progress by the same team, was a CRUD application at heart. Data out of the DB, a bit of manipulation, some input by users and services, and dump the data back to the DB.

Many out there disagree with my general statement that CRUD applications don’t make good candidates for DDD – but not only do I stand by that statement, but two of the other client’s projects nearing completion showed some of the real pitfalls of DDD. While previous projects had attempted to use parts of DDD to help simplify their applications, it had not all gone according to plan.

What Goes Wrong With DDD Over CRUD – The DB Domain Model

The first most obvious weakness of the systems that existed was the Domain Model. It had started with all the best intentions – there was a legacy database that acted as the primary store for the business, and many different systems used and updated this database. So NHibernate was chosen as a way to create a Domain Model over the database, to abstract the system away from the DB and to treat it as a legacy system.

Unfortunately one of the primary requisites of DDD is direct involvement and commitment from the business to the project and in this case that wasn’t part of the project. The team had to come up with their own Domain Model – and in doing so had fallen into the classic pitfall of creating a “domain model” that mirrored the legacy database. Now not only was there a legacy database, but essentially there was a legacy domain model too. A large amount of work was put into getting this NHibernate model right, and in hindsight it has probably consumed disproportionate resources to get this model overlaying the DB.

In addition, although at present it simplifies many operations against the DB, it does not abstract the code base from the database, it just acts as a heavyweight data access layer. Worse still, many of the “hacks” that had to be made to get NHibernate to work with the (frankly terrible) underlying database structure, had actually made the system very fragile, and fairly rigid.

Lesson to be learned: Do not drive your domain model from the database, drive it from the Domain Experts, user stories and use cases. Without significant commitment from your business to provide you with those experts, you are at risk of creating a significant amount of code, with no quantifiable benefit.

What Goes Wrong With DDD – Where is The Ubiquitous Language?

A somewhat related problem that is now obvious, again in hindsight, is that there was no clear attempt made to extract any kind of Ubiquitous Language out of the (non-existent) domain experts, nor from the business analysts. The language between the teams has fallen into the classic pitfall of a language confused across development teams, technical support teams, legacy systems, and business terminology that was not clearly defined nor understood.

Lesson to be learned: Get your language sorted out early on. Insist upon a basic glossary of terms at the very minimum, regardless of whether you do DDD or not. This will save countless hours of mistakes and confusion, and will get new developers and business people up to speed on the project in record time.

What Goes Wrong With DDD – Services Services Services!

I sat down early on with numerous team members and the architect who was in the unenviable position of making all this stuff work, and ensuring all the new stuff learned from previous mistakes and moved forward.

One of the things that became apparent to me was that while discussing architectures, the term “Services” was in frequent use, more than frequent to be honest, it permeated through everything. It was a term that business analysts and project managers were comfortable with, and perhaps that was the reason that “Services” was the terminology for decoupling things. From that it soon became obvious that Services of all kinds had become intrinsic to the current codebase, from Web Services, to Application Services, to Domain Services, and on and on.

Now, there is nothing wrong with “Services” per se, but interestingly when I suggested a new architecture for the new project, and discussed this with various people, I managed to explain the entire new architecture with almost no use of the word “Services”.

When I pointed this out, it soon became clear we could discuss a decoupled system, with little direct reference to “Services”, as fundamentally Services are an implementation detail, and have little important to the overall architecture of a system. Now instead of talking about Service being called, we can discuss messages being moved around the system, without any need to worry about what consumes them, or how they are delivered. The final architecture includes services for sure, but they are the implementation of a much more abstract concept.

Lesson learned: Technical language can become the norm, and then it tends to be the solution to every problem. Step back from things once in a while, and see if the terminology you are using is driving your architecture and design. Try to avoid using implementation terminology in relation to things like architecture, or it will drive the design.

So Where Are We Now?

The one really big thing that has happened over the past two weeks is that the introduction of some new blood to the team, along with some new ideas, has really revitalised the team. Everyone has new found energy. We also have a broad consensus of what didn’t go right on the last two projects, and the things that went well – we now have a strong idea of how to build on the strengths of the existing systems, and where we need to compensate for the things that didn’t turn out as planned.

Luckily the teams involved are all really eager to learn, and really keen to make each and every system better – and that alone will make a huge impact to how successful this project will be.

Well, this post has turned into a bit of an EPIC … and is far too long to continue without boring everyone to death … so I will post the concluding part soon. And with luck I may explain my opening statement … “We Are NOT Doing DDD”.

del.icio.us Tags: DDD,Domain Driven Design,Architecture

Job Satisfaction, And Making The World A Better Place

Jak Charlton — Sun, 29 Mar 2009 11:24:00 GMT

You know what we achieve with all our wonderful software patterns and practices? Not a lot on the whole.

We do fairly well financially out of it, and many of those who use our software do fairly well financially from it too. A select few of us write software that can have a small impact on the well being of others, but even then, it is in a totally detached way.

But how often can you say that your day at work has affected somebody else's life for the better, or made a really positive change to the world? How often can you look in the mirror and say my whole day was spent helping other people?

So, what the hell am I doing in the software game? Well, like I suspect many of us, I fell into it by accident. It was my hobby at school, I wrote games software and made a fairly comfortable (for a schoolboy) living from doing it.

Although my choice of O Levels (equivalent to GCSEs these days) were picked around becoming a doctor, and even though my mother had to argue with the school for weeks to get them to allow me to do three sciences, home and family circumstances at that time meant I never actually took most of my school exams. So with nothing in the way of real qualifications, I drifted into what I was naturally good at - technical support and writing computer software. My hobby turned into
my career.

Twenty odd years later, and I am more than comfortable with my career, I like to think I am pretty good at it. I also still often think, I could have been adding much more value to other people's lives if I had stuck with my original choice. Recently I met a lady who was a nurse, and I was totally inspired by her and how much satisfaction she clearly got from her work – she told me that I could still retrain even at this stage in life.

Now I expect you are waiting for some kind of development related conclusion here ... but unfortunately there isn't one. What there is, is a small realisation that I could probably be doing something more for the world.

So, I have started seriously investigating my options for a radical career change - and to retrain as a medical doctor. Now this isn't quite my swansong from development just yet. Not only do I have some major considerations to make, but I also have to put enough money aside to finance this shift, and after all that I have to succeed in an application to medical school, which in the UK is a very difficult task indeed, as places at our universities are very sought after, and the universities can afford to be very selective. I could even do one or two years getting the entry requirements, only to fail to get a place at medical school.

But, I have started sending off enquiries to various universities asking for their advice on the best route in, and perhaps it will pay off. It will be time consuming, financially very hard in the short
term, but it could be the most rewarding thing I could possibly do.

Ultimately, Job Satisfaction is About More Than Financial Reward

Whatever I eventually do, I have made a decision to get out and start giving back something to the world, maybe that will be with 6 years of medical training, followed by years of excessively long hours for very little pay ... or maybe it will be just finding ways to make people's lives better while continuing my career in software.

I encourage you all to find something positive you can give back to the world - the development community has some incredibly smart and insightful people - let's not waste all our efforts on arguing over whether we should invert our dependencies or not!

And if you want somewhere to start doing your small bit, can I suggest you take a read of The Girl Next Door by Bil Simser

Bloody hell that was DEEP!

Delay All Your Decisions

Jak Charlton — Sat, 21 Mar 2009 16:02:26 GMT

I sat down yesterday for a really interesting interview with a potential client. We discussed a variety of things, but one thing that stood out for me was a question around which message bus I would use and why.

Apparently there had been some discussions already, and the application being developed might need one in future. So the team had been discussing their options, after all at some point in the future the whole organisation could benefit from this and other applications could start to use this message bus to benefit everyone.

My response, and a rather unexpected one it seems, was “do you really need it?”, and “if so, what’s wrong with just using MSMQ?”

I’m all for simplicity, and YAGNI, which this decision would fall firmly under – but I’m also a great advocate of:

Delay All Your Decisions Until The Last Possible Moment

Although the team could have gone for a well established message bus, JBoss and RabbitMQ were mentioned as options – my view was that this was a decision they didn’t need to make now, and making that decision could possibly work to the detriment of this project, and possibly of the whole organisation.

While this project may well need a messaging system of some kind, the implications of picking one of these choices, or another like NServiceBus or MassTransit now were quite large.

Without any requirement from the organisation as a whole, picking a message bus and hoping the rest of the organisation would run with it too were small – something intended to work on this scale requires a lot more time, thought and consideration than was available, and putting it out for discussion at this point could delay the current project, potentially fatally.

Picking one of the options now for this project in isolation was a much better route, but the requirements for this application are as of yet unknown, and so any choice would be made on an arbitrary criteria, potentially going down a technology route for the sake of it, rather than a real need. And the more complex the solution picked for this application, the less likely it was going to fit with the organisation’s wider decision later on.

So my answer was simple … for now I would go with a very simple event broker in the application, and have that use the simplest possible method of messaging available right now (probably MSMQ). When the needs of the application developed, it would then be simple to bring in a more advanced bus system by just changing the event broker, or similarly if the organisation’s needs became clearer it would be easy to use their bus of choice.

This was a decision that didn’t need to be made at this point. Consider it a premature optimisation.

By taking a simple option now, and delaying the real decision to a later date, the team will gain flexibility on their immediate application, gain clarification of their requirements and of the organisation’s requirements as a whole, will avoid locking in to any technology that could be harder to back out of later – and most importantly of all – keep focused on delivering the project at hand, rather than get distracted by this decision.

When a decision stops being trivial, delay it until the last possible moment you can – you will always have more information by then, and will therefore make a better decision.

Welcome to Tuna – a Very Smart Fish!

Jak Charlton — Wed, 11 Mar 2009 16:39:10 GMT

Devlcio.us is proud to have Tuna Toksoz join us as our newest blogger … he really is one smart fish, and if you have been following his blogging on http://tunatoksoz.com/ then, you’ll be only too aware of his in depth knowledge of things like NHibernate and his involvement in Alt.Net Turkiye. He is also one of the most helpful and nicest guys I have come across in the InternetoSphere!

Looking forward to many great posts from him soon at: http://devlicio.us/blogs/tuna_toksoz

DDD: Invariants or Contextual Validation?

Jak Charlton — Wed, 11 Mar 2009 08:10:44 GMT

Greg Young made a good point to me regarding my last post about Validation, Consistency and Immutability, specifically around validation – even more specifically he thought I may have simplified it too far.

I was trying to cover the basics of the subject, but may have made the bit around validation a little fuzzy – over the series I have been deliberately taking baby steps so as not to lose anyone early. A lot of the confusion I see around DDD is largely around people jumping right into the deep end without the basic stuff clear first.

The last blog post had a very long section that I removed – it was entitled “Invariants” – but after I proof read it, it made absolutely no sense to me, so I figured it would make no sense to anyone reading it either. Turns out, by removing it, I may have made less sense too! I’ll try again. (disclaimer: suffering from really rotten head cold, so this may make no sense either – feel free to tell me if so)

Invariants

As the name says, an Invariant is something that cannot change – it cannot be varied.

Greg’s point was:

not to get philosophical but... consider Plato's ideals. What attributes of an object make it that object? Must be valid!

This is what I believe Greg likes to refer to as an invariant – or what I was trying to explain around using constructors to enforce “valid” entities.

This then splits our validation into two distinct camps – the invariants and the contextual validation.

An invariant is something that cannot be changed about our object – using our previous example, it must have a CustomerNumber, a FirstName and a LastName, and these must always be valid individually and when evaluated together – there can be no variance from that rule – and so we enforce that by only allowing setting these via the constructor or a strongly named method. No Customer can ever exist without these being valid.

The things that make our Customer a Customer are those attributes.

Contextual Validation

In my previous example, I used the concept that a Customer may have a Policy – let’s say this is an insurance scenario. I also said that the fact that the Customer didn’t have a Policy, it didn’t mean the Customer was invalid, it just meant we would have to have some kind of validation at the point we needed to have a Policy

This is what I believe Greg refers to as Contextual Validation – when we use the entity we may need to validate it is in a fit state for the specific operation we wish to carry out, but we shouldn’t have to validate everything about it.

In Conclusion

Greg said he is doing up a blog post of his own on the subject at some point in the future, which is good as the issue of validation seems to be a sticking point in DDD and in development generally. I definitely look forward to reading it. Keep an eye on his blog at Codebetter for it.

By defining the types of validation more accurately, we can try and avoid some of the confusion which I was introducing in my last post on the subject.

Invariants are the rules that cannot ever be broken about an entity, aggregate or VO

Contextual validation is a check that for a particular action, the entity, aggregate or VO is valid at that point.

del.icio.us Tags: DDD,Validation,Invariants

DDD: Sample Application First Steps

Jak Charlton — Mon, 09 Mar 2009 16:39:14 GMT

As mentioned in my last post, the DDD Parcel Service is opening up for business soon. Also as mentioned, the first thing I did was to grab my Domain Expert and start some initial conversations around the first areas I am approaching – the Booking service.

All of the documentation I am generating is to be included in the SVN (SVN public access to follow in the near future please be patient), so hopefully as the design evolves, and the UL and model change with it, this will be evident to some degree within that documentation. I’m going to get boring again, but the documentation is far more important in DDD than many of us Agilists would traditionally like – at least the resulting artifacts are, so it is important to me to show in the documentation why I am making code decisions.

The Tools

I am primarily using two nifty tools at the moment for capturing the information I need – the ever excellent MS Office OneNote 2007, and a new addition to my arsenal – Balsamiq

OneNote provides and excellent dumping group for notes and ideas as they get fired at me, and quickly allows me to form those into some coherent format.

But the true star of this process so far is Balsamiq.

User Interface Lead Design?

I wanted to validate the User Stories I had extracted from my DE, by showing how these could take form within the UI. To some this may seem like heresy, after all DDD is all about the domain right?

Well, I actually see DDD as being lead by a combination of the Ubiquitous Language on one hand, and the User Stories and Use Cases on the other.

Extracting the UL from the initial User Stories was part of the first task, but validation of these was far easier to do with some visual aid – so a mockup was needed. I could have done this in HTML, but luckily Balsamiq recently became a devlicio.us partner and I was keen to try it in anger on a “real” project.

Mockups Made Simple

The initial user story we worked out was around booking a parcel delivery, and after some discussion we got to:

The Customer wants to be able to send a Consignment , so they log onto the Client website to Book a Delivery:

They choose their Pickup Location
They choose their Destination
They choose their required Pickup Date
They choose their Delivery Window options
They choose the Consignment details
They choose any Special Instructions
They receive a Quotation for delivery if the system is capable of quoting, or a Callback for a Non-Standard Consignment
If they wish to proceed they click Confirm

Firing up Balsamiq was quick, and within a few minutes of drag and drop, and a few moments of cursing when I kept forgetting to lock items, I had a good starting point for the UI:

Less than 30 minutes later I had the whole story mocked up and ready to validate. Not unsurprisingly, it was really easy for my DE to look at these screens and envisage the process in action. This lead to a few changes in the UL, and a couple of extra questions arising – exactly the kind of validation that was needed. Had I done this in HTML or code I could have spent a good many hours doing something not half as representative.

If you want to see the initial modelling you can download them as PDFs here:

Kudos to both OneNote and Balsamiq for excellent applications to let me gather and organise the results of my initial conversations!

del.icio.us Tags: DDD,Balsamiq,OneNote,Modelling

DDD: The Beginning of My Sample Application

Jak Charlton — Sat, 07 Mar 2009 19:08:20 GMT

The Domain Driven Design Parcel Service is a small competitor in a hugely competitive market. With such competitors as UPS, FedEx and DHL, we are at a competitive disadvantage, and need a solution to cover our interactions with our customers, and to streamline our internal processes. This software will be at the heart of our expansion plans and needs to grow as the business does. Although we are currently only a national carrier in the UK, our long term vision has much wider reach.

The Domain Vision Statement (Evans page 416):

The DDD Parcel Service Booking Domain is to be responsible for Customer interactions. It should reflect the processes and practices our Customers want to use. This should reflect the relationship we want to have with our Customers, and easily allow them to deal with booking processes, managing their accounts with us, our discount and promotion schemes, and allow them to see a historical record of their business with us.

And So It Begins

I needed something concrete to start putting some of the concepts from this series into, to try and show how the patterns of DDD can be used in a “real” code base. In addition, I have spent a long time thinking about the specific domain for the example application, as it had to be something that had real meat to it, and real behaviour.

I could have gone down the shopping site route, but expanded it to be more like Amazon than a small one man operation – but as shopping sites have been done to death I had to find something else. Unfortunately the original example application for DDD also exists in a similar sphere to my choice. I did consider this carefully, and have weighed up many options.

The domain chosen had to be:

Easy to understand when you put yourself in the shoes of the business
Easy to understand when in the shoes of a customer of that business
Simple enough to not get swamped in technical details
Complex enough to show a number of aspects around the things DDD brings real benefits to
Have real behavioural process rather than just another database and a pretty UI

In addition, I had to be able to deal with the domain with little friction – although it may have been fun to learn some of the domains that various people suggested to me (including the Oil Drilling Industry!) – I only have a limited amount of spare time, and my girlfriend has a big priority on most of that.

The DDD Parcel Service fulfilled all of my simplicity and complexity requirements, it is a pretty common domain to most of us in everyday life, at least from the Customer perspective. It also happens to be a domain that I have touched on in the past from a business perspective.

It happens that with my choice of a Parcel Service domain I had easy access to a Domain Expert of sorts – my girlfriend has a long family history in the haulage business, and on a daily basis manages the haulage for a customer just like those that the DDD Parcel Service would have.

This Morning’s Progress

Despite both myself and my girlfriend suffering bad colds, I pinned her down for 20 minutes to start the first session on extracting some of the user stories for the DDD Parcel Service

I then started hammering away at a few screen mockups in Balsamiq … and I think on that point I’ll start a new blog post for tomorrow or the day after.

So Where Is The Code?

Don’t be in too much of a hurry! I have started a project, and have the core framework starting to form up. But more importantly to me, and to DDD is the design, the extraction of the UL, the agreement of the User Stories, and the way we get to the code. Over the next week or so I will try and start turning the initial stories into a beginning code model – and at that point you can all rip it to pieces!

Technorati Tags: DDD,User Stories,Balsamiq

Make Your Own RSS Feeds With Yahoo Pipes

Jak Charlton — Fri, 06 Mar 2009 15:11:14 GMT

When setting up the DDD series on http://dddstepbystep.com I wanted to bring it various information from across the web, to try and consolidate a lot of Domain Driven Design info all in one place.

My first thought was I would setup a blog, and whenever I found something that I thought was a good fit, I would copy a summary to the new site, and link back to the original piece. This was a great idea, apart from the fact that it was going to turn out to be horribly time consuming.

The next step was finding a better way of doing this, hence one of the reasons I hose to host the new site in Community Server – it has a “Mirrors” setting for each blog, where you can add any number of RSS feeds, and it will bring in a summary for you every time the job runs.

After a lot of messing around getting CS set up, I found one of the major limitations of CS is the “Mirrors” functionality … although it is good at actually bringing in the feeds, it has no way of filtering, combining or correlating feeds. It also has an arbitrary way of detecting duplicate entries, which appears to be based solely on the title – which is less than simplistic.

After a bit of searching and trying various options, I came across Yahoo Pipes – this has got to be one of the coolest tools on the web right now.

With Pipes I was able to set up a complex set of feeds, followed by a series of transformations that allowed me to sort out all the discrepancies that CS was unable to deal with.

The “DDD on the Web” Blog Feed

Firstly I was able to combine half a dozen feeds of blogs I know regularly have great information on DDD subjects. This was a simple matter of giving the Fetch Feeds source the URL of each RSS.

Then I needed to filter out any blog posts by myself – as one of the feeds is coming from devlicio.us I didn’t want to duplicate my posts on each site, as I post the full blog directly to the new site as well as here. With a quick drag and drop of the Filter operator, my posts were gone.

Next I wanted to remove all the posts that didn’t mention DDD or Domain Driven Design – another Filter in place and I was closer to what I needed.

The Unique operator allowed me to then remove any duplicate posts based on their content – the CS one would have allowed duplicated content if the title was different.

Next I wanted to add the original author name into the title … to turn “This is my Blog Post” into “This is my Blog Post by Dave D Spatch”. Things then got a little tricky. As I was bringing in feeds from various sites, and they were inconsistent in their naming, the author fields were often coming through in the wrong place. Using a combination of the Rename and Regex operators allowed me to sort this out and provided me with a much better title. A last Regex allowed me to replace a number of “username” type fields with the full names of those I knew – so for example I replace “colinjack” with “Colin Jack”.

And finally – we get a single feed that I pass on to Community Server for it to import. The last step was a simple cosmetic change, I changed the URL from some horrible id string, a nice human readable one: http://pipes.yahoo.com/dddstepbystep/ontheweb

All of which is now brought in correctly to the blog at: http://dddstepbystep.com/blogs/ontheweb/ where I can still manually add anything of interest the feed has missed.

The “DDD at Yahoo” Blog Feed

I wanted to also bring in the feed from the domaindrivendesign group at Yahoo – where a lot of general discussion around DDD takes place.

This had some similar elements to the other Pipe, but also required me to work around the limitation of Community Server I mentioned earlier – it detects a duplicate item based on the title alone – and skips it if it finds an existing match.

As the DDD Yahoo group has a lot of “Re: Your Subject” titles, CS discards all but the first it finds if you use the RSS directly.

Pipes again allowed me to resolve this, by appending the author to the beginning of the title, and adding the time to the end to the title, I can get almost uniqueness. The title looks a bit ugly, but I couldn’t find a better way to fool CS into doing it right.

In Conclusion

Yahoo Pipes is a superb tool – it can be a bit tricky to understand at first, and some things still seem convoluted to me … but it has allowed me to work around some tricky limitations with relative ease.

If you find a need to make your RSS feeds just a little better – then take a look. Better still, it can work with data from all over the web, and turn that into an RSS feed for you.

del.icio.us Tags: DDD,Yahoo,Yahoo Pipes