This month, Nathan Bain and Anand Ramdeo share their experiences of working as Test Engineers within Agile development environments. They will pass on some tips they have learned along the way, discussing how they used tools such as Selenium and FitNesse. The talk is on the 22nd July, starting at 6:30 at Skills Matter. It’s free, but you have to sign up in advance for capacity planning. See http://skillsmatter.com/event/agile-testing/agile-testing-tools-and-approaches for more information.

Problem 1 entails figuring out the sum of numbers divisible by 3 and 5:

If we list all the natural numbers below 10 that are multiples of 3 or 5, we get 3, 5, 6 and 9. The sum of these multiples is 23.
Find the sum of all the multiples of 3 or 5 below 1000.

I thought it would be interesting to see if I could solve this in Groovy as the solution clearly deals with iterating over a series of numbers — and as everyone who has played with Groovy (or some other dynamic language for that matter) knows, iteration is a blast, baby!

Accordingly, my first pass yielded the following code:

def sum = 0
(1..<1000).each{ if((it % 5 == 0) || (it % 3 == 0)){ sum += it }
}
assert sum == /* do it yourself, baby! */

Note that I’m using Groovy’s exclusive range feature to easily iterate over all numbers less than 1000. I then proceed to use Groovy’s it variable, which represents the current value in the iteration (i.e. 1, 2, 3, etc) and test the instance with Java’s modulo operator (that is, modulo returns the remainder of division — 10 modulo 5 is 0, but 11 modulo 5 is 1). If there is a match, I add it to the sum variable. This is programming 101 and brought me back to the Age of Aquarius, baby!!

That was easy enough; however, I wanted to see if I could do away with the sum variable by using one of Groovy’s many hip magic methods attached to collections. Thus, my second attempt leverages Groovy’s findAll method, which permits putting a condition in the resulting closure, which then returns a collection meeting that criteria. Accordingly, with the returned collection of numbers meeting my criteria (that is, any number that is divisible by 5 or 3), I then have to issue the sum method like so:

def val = (1..<1000).findAll{(it % 5 == 0) || (it % 3 == 0)}
assert val.sum() == /* you gotta figure it out yourself, man! */

Groovy’s sum method is straight forward — it takes all the values in a collection (presumably add-able) and adds them up!

Admittedly, the second example is a bit harder to read (at first glance, that is) but it sure is a bit more aesthetic than the first brute force bogue example, isn’t it?

Apache Sling in a few words

“Apache Sling is an innovative web framework that is intended to bring back the fun to web development. It uses all those nice cool and new technologies that make up a state-of-the-art framework. This is Apache Sling in five bullets:

• REST based web framework
• Content-driven, using a JCR content repository
• Powered by OSGi
• Scripting inside, multiple languages
• Apache Open Source project

Some size indications of the project

• 40 Maven modules
• 70,707 lines of code
• 731 Java classes
• and 23,043 lines of Javadoc

The strengths in terms of quality

• A project that you get and compile with no difficulty by running two commands :
       1. svn checkout https://svn.apache.org/repos/asf/sling/trunk/
       2. mvn clean install
  This sounds like an evidence but is not always the case :-)
• Amongst 130,172 physical lines, only 0.9% are involved in a duplication
• 46.4% of public API are commented with a Javadoc block

The weaknesses

• Only 9% of the source code is covered by 338 unit tests
• Average cyclomatic complexity by method (excluding getters and setters) is greater than 3 (3.2).
  That is kind of a warning saying “your methods are taking too much responsibilities and should be re-factored”. This warning is confirmed by others metrics : 394 methods have a complexity greater than 7 and 86 methods have more than 50 statements. What is true at method level gets also partially confirmed at class level as 60 classes have a Fan Out Complexity greater than 20 (The number of other classes referenced by a class)

Bad programming practices that should be improved

• 198 times, method parameters are reassigned in the core of the method
• 68 times, local variables are defined and hide class fields
• 28 times, NullPointerException are thrown when an IllegalParameterException would be more suitable

Potential bugs that should be quickly analyzed

• Correctness - An apparent infinite recursive loop : there is an apparent infinite recursive loop in org.apache.sling.scripting.jsp.jasper.runtime.JspContextWrapper.include(String, boolean)
• Multithreaded correctness - Unsynchronized get method, synchronized set method : org.apache.sling.scripting.jsp.jasper.compiler.JspRuntimeContext.getJspReloadCount() is unsynchronized, org.apache.sling.scripting.jsp.jasper.compiler.JspRuntimeContext.setJspReloadCount(int) is synchronized
• Multithreaded correctness - Method calls Thread.sleep() with a lock held : org.apache.sling.event.impl.JobEventHandler.runJobQueue(String, JobBlockingQueue) calls Thread.sleep() with a lock held
• Malicious code vulnerability - Field is a mutable array : org.apache.sling.jcr.webdav.impl.servlets.SlingWebDavServlet.COLLECTION_TYPES_DEFAULT is a mutable array

This analysis was done with the intention of giving a synthetic overview of the current state of the project. Where should you start from if tomorrow you wake up with a single idea in mind : “Improving quality of the Apache Sling project !” ?

• With respectively a cyclomatic complexity of 428, 385 and 343, classes Generator, Parser and XMLEndoginDetector should be first refactored. With no surprise, the Generator.java file has the greatest number of duplicated lines (154) and rules violations (109)
• With its 43 cyclomatic complexity and no unit tests, the method ModifyAceServlet.handleOperation(..) is what we call “a crappy method” :-)

More information on the code quality of the project is available on Nemo.

June 30, Harrisonburg, VA: Shenandoah Ruby Users Group, Social Media 201 (Jason Hull)

July 23, Las Vegas, NV: National Veterans Conference, Leveraging Web 2.0 Applications for Productivity (Jason Hull)

August 27, Chicago, IL: Agile2009, How to Sell a Traditional Client on an Agile Project Plan (Arin Sime)

October 21, Cambridge, MA: Software Test & Performance Conference, Setup and Testing with Continuous Integration (Eric Pugh)

If you’re going to be at any of these conferences or meetings, give us a shout! We’d love to meet up with you!

While traditional software methodologies have the tendency to segment groups into logical units of responsibility, agile teams favor a more collaborative approach that facilitates working along side various people and groups containing differing skill sets and expectations. That is, for example, the client begins to take responsibility for elaborating expectations through a working effort to understand how the process unfolds. What’s more, by working closely with a client, a team can better understand what is needed and why. As opposed to a test document fashioned from a bogue requirements document (often times done by various teams working separately), leveraging a story in a collaborative manner ensures all parties are effectively on the same page. Thus, the customer (or some proxy there of), development, and QA are working together in the process of constructing stories and indeed leverage the same stories for all related assets.

Because it’s their bag, when Agile teams embrace stories as a fundamental means as to conveying requirements (i.e. features) along with a means for validating them, a natural tendency is for all parties to begin to own quality. That is, since everyone is aware of how to validate a feature, everyone begins to practice validating such features. In essence, a story facilitates leveraging both TDD and functional testing (which are also both automated and exercised via CI) .

What’s more, when the whole team both works together off of the same artifacts that are continuously verified, everyone can better understand the meaning of “done” — that being the question “is feature x finished and ready for the client (or clients) to have?” Of course, if everyone has utilized the same artifact (i.e. a story!) for both building and verifying a client’s expectations, then it is quite easy to check if the feature is done.

Thus, the whole team approach that leverages stories:

• ensures all hip parties see and understand the “big picture” of a particular project
  • that is, why someone wants an application (and its related features) built, which in turn further facilitates whole team ownership (along with more effective testing!)
• helps teams figure out the definition of done

Obtaining these benefits is a matter of providing a means for all parties on a team to collaborate, which is often found in leveraging a story. Plus, visibility into overall team progress as facilitated via CI and automation. The whole team approach also meets the original goal of software testing — that being the assurance that high quality code is delivered to interested parties. Teaming also reduces the tendency to throw up barriers that often plague efficiency.

Agile testing boils down to two fundamental changes, baby, that differentiate it from traditional software testing — both the notion that testing isn’t a phase but a whole life-cycle approach and that the entire team works together towards shared goals. Agile testing complements a highly efficient software development life-cycle and ultimately produces better software faster.

• Let’s say you have built a commercial tool (PC-Lint, Simian, SAP ABAP Code Inspector … ) or an open source one (Testability Explorer, Crap4j, Clang Static Analyzer…) or let’s say that you think a new axis of quality analysis would be more pertinent. By developing a plugin in Sonar, you kill two birds with one stone : you immediately have access to a large community of users and you benefit from the built-in functionality (TimeMachine, trends, consolidation…)
• You manage a consulting company and are building a package around quality. By building this package around Sonar, your client gets basically the tool for free and what they are going to pay for is what you have to sell : your expertise on quality

I believe that code quality management is going to become a commodity in the medium term as is continuous integration today; and I do hope that Sonar is going to play a central role in this democratization move.

Luckily, dragging my project into the present wasn’t difficult. Along the way I also added my essential testing libraries: Hamcrest for matchers and Mockito for mocking (well, stubbing, but that’s another story). For posterity’s sake, and for others that share my testing tastes, here’s how it’s done.

Requirements

I’m assuming that you have Maven 2 installed already. If not, it’s trivial to:

1. Download the latest (2.1.0 at time of writing); and
2. Install it according to the instructions provided.

You can check if you have Maven ready to go by running:

Bootstrap

With the lack of an available alternative, I found it easiest to start with the default archetype. To create a new project, run something like:

Remember that if you’ve just installed Maven it will take this opportunity to download the internet. Be patient. If you’re new to Maven, you might also want to check out the 5 minute guide which walks through this in more detail.

Check that your bare application has been created:

Add the Dependencies

Next you need to update the project POM to add the testing libraries as dependencies to your build. This involves three changes:

1. Removing the default dependency on JUnit.
2. Adding new dependencies for TestNG, Hamcrest and Mockito to the “test” scope.
3. Configuring the compiler to accept Java 5 source.

The last step is necessary as the Maven default setting assumes Java 1.3 source, which apart from being ancient doesn’t support goodies such as annotations that are required for the new testing libraries. Your updated pom.xml file should look something like:

I’ve used the latest available versions of each of the libraries in this example — tweak them to suit your current reality.

Update the Sample Test

Now you’re ready to try updating the sample test case to use the trinity of TestNG, Hamcrest and Mockito. The easiest way to do this is to get Maven to generate a project for your IDE, e.g.

or:

Fire up your chosen IDE, open the AppTest class, and edit it to exercise all of the dependencies:

What are you waiting for? Try it out:

If you got this far, then everything you need is in place. Now you just have to … implement your project!

Extra Credit

If you poke about a bit, you will also find that the maven surefire plugin, which manages the tests, generates some reports by default. Along with HTML output, it also produces a JUnit-like XML report at:

This report is ideal for integration with a continuous integration server (in my case Pulse, naturally, but many will support it).

Happy testing!

• Global dashboard
• Violations by categories
• Hotspots

A full example of the report is available here.

What about the future? The future is Team Workbook Report, the second report type that will include most deep data as violations per class and line, graphics and more.

The author : Antonio Manuel Muñiz, www.gmv.com (Blog, email)

The MSMIT program at McIntire is a 12 month program for working professionals, where we typically meet for 3 long days a month, with group work, projects, and lots of reading and research in between the monthly sessions. Module One (or Mod1 as we call it) is different however. This is our full time 2 week residency at UVa, where we study various foundational IT topics in class for 8 or 9 hours. Topics like: Database design, Data network design, UML modeling, the Zachman framework, Business strategy, Web 2.0, Datawarehousing, IT security, and more.

But that’s just during the day. At night we work on a group project, in addition to some homework assignments, guest lecturers (we had some great speakers visit), and social events to network with our classmates.

The group project for Mod1 was to come up with a Web 2.0 initiative for a health insurance company, design the product, research and estimate its business value, and prepare a presentation for the class. At the end of the two weeks, we had two things to deliver. First, we had to turn in a complete Zachman framework design of the product. Second, we gave a presentation to our classmates as if they were the Board of Directors of the health insurance company and try to convince them to fund our project.

I had the pleasure of being on a great team. At the beginning of Mod1, each group had to complete some team building exercises, where we decided what the ground rules for our group would be, what our group and individual goals were, and how we wanted to operate as a group. I found this to be a more valuable experience then I expected, and I plan to use it more often in projects at work.

One important decision that our group made right from the start was to use a lightweight version of Scrum to organize our team. Our team’s goals were to be very efficient, and to focus on the quality of our work, not the quantity of time we spent on it. Since two of our team members had previously used Scrum, and a third is a rugby player (Tim), using Scrum was a natural fit.

We ran 1 day sprints over the two weeks. We didn’t keep a burndown chart, since that would be too much overhead. Our planning sessions basically were to update a list on the wall with everything we were going to do that night, and then go around the team and let everyone pick their own tasks until all tasks had been assigned. We didn’t bother with estimates on those tasks since the granularity of the sprint was already down to just one day.

At our next meeting, we would do a quick stand up to update the team on where we stood on our assigned tasks, and then we usually did a demo of our tasks. This “lightweight Scrum” helped keep us very focused on our work and the tasks assigned to each of us. At OpenSource Connections, we did a similar thing for the Rails Rumble last year, which is a 48 hour coding competition. During the Rails Rumble, we would plan sprints of 2 or 3 hours each, at the end of which we would demo our progress to each other. It was lightweight enough to work in a very tight timeframe while still keeping just enough process that we all stayed focused on the job at hand.

The other efficiency thing we did was to “time box” every discussion we had. Many parts of the group work involved design discussions about what we wanted to do for our projects, debating the pros and cons of it, reviewing each other’s work, etc. To keep these discussions from going on all night, or wasting our time debating things in detail that might not even end up in the project, we would put a time limit on everything.

Every day we tried to rotate a facilitator role, and that person’s job was to keep the discussions going smoothly, and they were usually the person who would pipe up with “how long do we want to spend on this topic?” We might set a time limit like 15 or 30 minutes, and then Adil would set the timer on his iPhone. When the timer went off, we would cease debate on the topic and try to make a decision. If necessary, we would extend the time, but the goal was always to stay within in a timebox so we wouldn’t be up all night and could get some rest (or at least head over to the bar!).

We had a lot of different personalities on our team, and lots of different perspectives. But we got along great as a team and I think the agreement on ground rules and the use of Scrum helped out a lot with that.

I think the way we came up with ideas worked pretty well too. We started by encouraging everyone to write any idea they had on yellow post-it notes, and stick it up in one corner of the wall (you can see Adil pointing at the post-it notes in the photo below). It didn’t matter if it was a good or bad idea, the point was to generate as many as possible and get them on the wall. Similar to the way people propose sessions at an open spaces conference, each person would read their idea as they posted it to the wall. No criticism or debate was allowed at that point.

The next day we grouped the post-it notes together by ideas that were similar or could be easily combined together into the same project. It was clear that we had a lot of ideas around mash-ups, since that was the largest number of post-it notes. But we didn’t choose the idea based on quantity.

Arash had drawn a map on one whiteboard of the goals, strategies, and problems at the health insurance company (see pic below), and we began to move the post it notes onto the whiteboard, lining them up with what goals/problems they addressed.

At that point we finally started voting on what idea to go with. Everybody picked their 3 favorite ideas, and the three with the most votes were the finalists. We debated those three ideas in more depth, bounced ideas off Prof. Grazioli, and then picked our project idea.

Although it sounds a little complex, I liked this process because it encouraged us to all be creative, to consider all ideas, to map those ideas to the goals of the project, and ultimately to create a consensus around the final idea. I have a tendency to push for my favorite ideas, and this process helped make sure I didn’t force my ideas on the team, and ensured that we carefully and creatively considered all the ideas.

In the end, our team ended up proposing an incentives program based in part on an iPhone app that would record your exercise and upload that data to the health insurance company, which would then convert that data into “health points” for use in contests with fellow employees to encourage healthy lifestyle changes.

All the presentations were given on Friday, and the professors split the class into three different classrooms of 4 or 5 teams each (there were 13 teams total). Each team presented to their classroom, and then the room would pick which team should “advance” and be one of only three teams to present to the whole class in the afternoon.

Our team was really honored to be picked as one of the final three teams to present to the whole class. It was a lot of fun. I have to also tip my hat to the other two final teams since their presentations were not only excellent, but I was also very impressed with the depth of financial analysis they incorporated. (I’m looking forward to future modules where I’ll learn the finance skills to be able to do the same since finance is not my background).

We kicked off our presentation with a video to illustrate the idea of our product. We had a lot of fun producing it, and I think our classmates enjoyed it too. You’ll have to pardon the poor resolution - it was my first time using iMovie and a flip camera, and I didn’t get the conversion right.

Just for fun, I also took a few clips from our filming and made an “out takes” video. We got to show the outtakes to the whole class too after our presentation, which I hope added a little playful humor to the end of Mod1. (Prof. Grazioli showed some very funny retrospective slide shows that were the real hit of the afternoon however!)

Mod1 was a great experience. I advanced my IT skills, teamwork and leadership skills, but perhaps most of all made some great friends through the shared experience of great professors, very interesting classmates, and of course sleep deprivation. I’m looking forward to Mod 2!

Team 11 was: Tim Bucher (Booz Allen Hamilton), Adil Qazi (Freddie Mac), Arash Sadati (International Monetary Fund), Arin Sime (OpenSource Connections) and Mark Widener (Virginia Air National Guard).

This is a summary of my presentation titled “DDD in a distributed world” at the DDD Exchange 09 in London. You can also download the slides accompanying this presentation.

One of the most important building blocks of DDD that can help in distributed systems are aggregates. Unfortunately, at least judging by the discussions that I’ve had with client teams over the last few years, aggregates seem to be one of the most underrated and underused building blocks of DDD. I’m probably as guilty as anyone else of misusing aggregates and it took me quite a while to grasp the full potential of that concept. But from this perspective I think that getting aggregates just right is key to making a distributed system work and perform as expected.

Aggregates are about units of consistency – not about pointers

Most folks think about aggregates in terms of pointers, such as that objects outside of an aggregate should only reference an aggregate root and not any objects inside the aggregate. Although very useful in general, this rule is not the whole story of aggregates, but that is where most discussions end. Aggregates are also perceived as very technical in nature, almost something that should be left out of the discussion with business experts, similar to repositories or factories. I think that this is a mistake. Aggregates represent a very clear business domain aspect that should definitely be discussed with domain experts. I found it much more important to focus on the fact that an aggregate is a unit of consistency from a business perspective.

We can look at an aggregate from a business perspective as a units of consistency in our problem domain. Something that in a business sense always needs to be perfectly coordinated and consistent, something that is only complete as a whole. Eric Evans calls this a “meaningful whole”, suggesting that an aggregate itself has a meaning separate from the meaning of its parts. As such, it is a genuine business domain concept, not just a technical guideline about references. And in order to find correct aggregate boundaries we shouldn’t be concerned that much about referencing code and class encapsulation. This is more an exercise of identifying business requirements for consistency.

From that perspective, aggregates have a clear impact on processing models and implementation of the system. An aggregate is a natural unit of distribution because we probably need all the information in an aggregate to meaningfully process it. Whenever we send information down the wire, the whole aggregate needs to go. It also means that we need to ensure that all changes to a single aggregate are saved correctly as part of a single transaction. Once we have done it, we genuinely have a meaningful piece of work done. This potentially gives us freedom to work on different aggregates in parallel or in separate transactions.

When these forces are aligned, we’ll get a very nice distributed system design and an implementation that performs nicely. When these forces are in conflict, we should stop and analyse the system instead of just ploughing away with the code. The conflict may come from either the technical or the business perspective but having these concepts explicitly defined gives us a good framework to spot something smelling dirty and fix the problem before it becomes too big. That is why aggregate models provide some great hints about distributed processing.

Hint #1: If you don’t aggregate enough, even the fastest computers will spin their wheels in vain

Latency is one of those classical problems of distributed systems. It is a very elusive concepts that an average programmer just ignores, because it seems as a networking issue that should be solved by networking people. I remember a guy who was building his own messaging solution and proudly boasted about its performance, demonstrating it by running both the server and the clients on his machine. When we deployed it properly and put one system in the UK and another one in the States, the whole thing fell over almost instantly.

Think of it this way: to send an page by post you need to find an envelope, put a stamp on that and put it into a post box. There’s also one less visible step there that most people don’t take into account – walking out of the house and going to the post box. Walking outside to a post box is where the latency kicks in. It’s not something people see integral to the process, but it nevertheless exists. Say you wanted to post a contract that has twenty pages. Doing the whole routine for every single page would instinctively feel wrong and no sane person would do that – we would just put all the pages in a large envelope and stick a single stamp on that, then take all that the post box at once. Yet I find people doing the equivalent of posting each page separately in software over and over again, making the system walk to the post box twenty times instead of once and paying for twenty virtual postage stamps. And this is where aggregates start to kick in. The whole twenty page contract is an aggregate and that very fact that is an aggregate should suggest that it should be shipped together. It is a meaningful whole so whoever needs a page probably needs the entire thing.

So I suggest the first rule to help distributed systems with aggregates: Ship entire aggregates straight away to avoid latency.

Hint #2: If you aggregate too much, baggage becomes a problem

The second classic problem of distributed systems is serialization performance. Serialization and deserialization are often among the biggest performance hogs in a system as they rely on reflection, IO and traversing complicated object graphs. This is again where aggregates can help, but the issue is a bit more subtle.

We need to keep an eye out for conflicts between the intended processing model and the business view of aggregates. For example, in a classic account system with accounts and transaction histories, it seems perfectly logical to have the account history as part of the account aggregate.

Say we wanted to process transactions for an account on different machines. Sending huge object graphs over the wire all the time makes no sense technically, so shipping the whole history when we send the account would kill the system performance. There we have a conflict between a business view of an aggregate.

Instead of just ploughing away with the code and breaking the aggregate apart for this one occasion (which would lead to the corruption of the design eventually), I suggest pausing and thinking a bit more about this conflict. This is a very subtle trap. First of all, there are no universal models and the model along with the aggregate structure only works within a particular set of scenarios, so stuffing the account history to be part of the account aggregate may or may not make sense depending on how we are going to use the accounts. When there is a conflict between the processing model that we wanted to implement and the aggregate model suggested by the business, that might point to better ways to design the system.

The rule that the whole aggregate gets shipped together would make us ship the whole history each time we send out account information and makes absolutely no sense. On the other hand, being able to meaningfully operate on just a few pieces of information conflicts with the business definition of the aggregate.

An option would be that the history is not really be part of the account aggregate – maybe it is just a list of domain events that record what happened, or maybe they form part of another statement aggregate. In that case, the model gives us a real argument to talk to the business about statements being something that can now get updated asynchronously as it is not part of the main account aggregate. This is why you get the correct balance but not all the statements on in internet banking applications. Inserting a new transaction record might require locking the sequence generator or a table in some database systems, so it costs a lot. If the transactions can be asynchronously updated then we can significantly reduce locking and improve the system performance. This model gives us distributed processing but clearly points out that the history is not going to be 100% consistent with the balance all the time. It would require a recognition from the domain experts that the history does not belong to the unit of consistency of an account.

On the other hand, if we don’t get the confirmation from the business about that, the fact that all of it has to be part of the same aggregate tells us that distributed processing of a single account is not where we want to go. We need to find a different solution, possibly a distributed master-worker with worker machines owning accounts and the master routing tasks to be processed on the appropriate machine.

So the second rule I’d like to suggest is: If you don’t need it all, it is probably not an aggregate and you can improve performance with asynchronous updates.

Hint #3: If you don’t aggregate enough, things are going to be broken just when you need them

This relationship between synchronous atomic updates and aggregates works both ways. Without properly defined aggregate boundaries in the model, we might actually be breaking up transactions into pieces that do not make sense individually. If there are no aggregate boundaries, there is nothing to tell us what actually is a consistent update from a domain perspective and we’ll probably form transactions around requests. This might lead to a lot of unnecessary code to handle technical “edge cases” which aren’t really business exceptions at all.

For example, a trading system might require customer’s credit card details for any meaningful operation. Because of web design constraints, the registration process for customers on the web site might be divided into several steps, where they would first fill in the personal details and then put in the payment details on a separate screen.

If we don’t capture the fact that the payment information are actually part of the customer aggregate then the normal way to implement this would be to form the transactions around requests – posting after each screen. That would allow the system to create customers with no payment information (when they close the browser or just don’t complete the registration). These strange cases will start popping up in reports and we’d have to implement workflows to handle these customers, preventing them from accessing functionality that requires payments etc.

If we do consider both parts a single aggregate, the rule that we should ship entire aggregates (even from the web site to create the customer account), leads to the conclusion that the data should be collected from both screens first and sent at once, instead of sent after each individual screen and processed in the database.

Because aggregates are units of consistency, in order for them to stay consistent we need to keep all steps in an update of an aggregate together. Aggregate updates need to obey the samurai principle: either return successfully or not return at all. Without this, we are opening the doors for data corruption – not in the disk failure sense but in the business sense. The technical edge cases that don’t exist in the business simply go away, so we don’t have to handle them or maintain code to work around them.

Atomicity of transactions fits into aggregates really nicely and this is where transactions also come in. If we think of aggregates as units of consistency, that allows us to design the transactions in our system to work on a single machine as we can ensure that all the information that should be processed synchronously within a transaction resides in a single place. Aggregates are natural atomic parts of transactions.

The third rule I suggest for improving distributed systems is: We should form our transactions around business units of consistency rather than technical requests to avoid data corruption and simplify processing. The entire aggregate should be processed synchronously within a transaction.

Hint #4: If you aggregate the wrong things, deadlocks galore

Locking resources in distributed systems is also one of those invisible issues that can just kill you if you don’t get it right. Distributed systems are typically built to process more than a single machine can – to increase the overall system throughput. Yet if critical resources are locked too much and processes often have to wait for other processes to release those locks, spanning to multiple machines might actually give you worse performance.

One example that we had in a recent DDD Immersion workshop, changed a bit to protect the innocent, is a scheduling system that involves event organisers and events. The unit of consistency in a business sense was an event organiser, so that was the aggregate by design. At some point, the system was integrated with an external notification source with messages about changes to events. A message might contain updates for hundreds of events for different organisers, postponing, rescheduling etc. The business wanted a notification message to be processed within a single transaction, regardless of what it contains. After the change, the system performance seriously deteriorated as a single notification message was locking hundreds of organiser objects.

This is another example of the processing model conflicting with the business model of aggregates. If the organiser and its events are units of consistency, then we should be able to update one organiser entirely and not update the others and the system would still be consistent. If this is confirmed by the business (which is more likely than not), then we have a good case against the requirement to process the entire message in a single transaction. We can take the message, store it and commit, then break it apart into chunks related to individual organisers in another transaction and commit that, and then process individual chunks in separate transactions. This model also allows us to process individual organisers in parallel so the business will probably see the final effect sooner than with a single transaction anyway.

The fourth rule coming out of this is : When locking, lock aggregate roots. Challenge requests to lock more than one at the same time.

Reinforcing ideas

Aggregates are a curious pattern as they clearly have both a business domain and a technical side. These two sides of aggregates confirm and reinforce each-other. That is what makes them so useful for modelling and designing distributed systems. Once we know what makes a conceptual unit of consistency, we have defined the clusters that need to travel down the wire or be processed as a single unit. If the required processing model contradicts that, then we need to revisit the aggregate definitions or the processing model.

Once we draw the lines of the aggregates and think of them primarily as units of consistency we can have a meaningful discussion on what implementation model suits our required set of scenarios. Aggregates are natural choices for some of the most tricky issues of distributed systems, including locks, transactions, asynchronous processing, serialization and latency.

If we want one but not the other that probably means that we don’t have the correct aggregates boundary, or that we have unrealistic expectations. And spotting this early on is one of the greatest powers of aggregates as a concept – we don’t have to burn our fingers by spending months developing something that simply is not going to perform or satisfy requirements.

I took notes during other presentations and will post them soon. See earlier DDD exchange posts and subscribe to the RSS feed to get notified when the new ones come online.

I downloaded flux2/beta from http://theescapers.com/ and I must say, nice toolset. It has a very developed plugin set, including roundcorners, jquery, mootools and can also be used for rails developments.

So to summarize: it looks every bit promising, so why not enter the competition. Here’s how I see it: