Jimmy Bogard

MVC Best Practices

bogardj — Wed, 28 Oct 2009 05:45:26 PDT

Simone has a great post (as usual) on 12 ASP.NET MVC Best Practices:

Controller:

1 - Delete the AccountController

2 - Isolate controllers from the outside world

3 - Use an IoC Container

4 - So NO to “magic strings”

5 - Build your own personal conventions

6 - Pay attention to the verbs

Model:

7 – Domain Model != ViewModel

8 – Use ActionFilters for “shared” data

View:

9 – NEVER use code-behind

10 – Write HTML each time you can

11 – If there is an if, write an HtmlHelper extension

12 – Choose your ViewEngine carefully

This is a fantastic list, and of course I have my own additions/suggestions :) First, I’m not a fan of action filters for shared data. If we’re doing strongly-typed views, using action filters for shared data puts us back into the magic string land, where we have to either use inheritance in our ViewModels, or the dictionary part of ViewData. I don’t like either approach, I’d rather go with something like RenderAction to truly enforce SRP in my ViewModels. Not everyone appreciates the elegance of this approach, but I’m sure they’ll agree eventually.

I also don’t like writing an HtmlHelper extension just because I have logic in the view. For a half-decent view engine, it shouldn’t be that problematic to have view logic in the view. HtmlHelper extensions enforce the Helper object anti-pattern – a bunch of procedural logic hanging off one static class. Instead, we went the route of building intelligent input builders for input elements, and only really using HtmlHelper when we want to eliminate duplication between multiple views. I see these constructs:

HtmlHelper
Partials
Master Pages

All as means of eliminating duplication in our views, but not much more. If there’s logic in a view, that’s fine by me as long as it’s not duplicated. It’s not my fault that C# looks downright bizarre mixed in with HTML markup, but that’s what other view engines are for.

Finally, if there was an award for How Not to Design a Controller, the AccountController would be the runaway champ. I still can’t imagine why it’s necessary, and the idea of portable areas would be a better fit for its functionality than the Thing We Always Delete with Extreme Prejudice.

MVC Web Testing Strategies – verifying content

bogardj — Tue, 27 Oct 2009 18:51:46 PDT

Some of the questions during the C4MVC presentation concerned how I liked to locate data displayed in the rendered HTML for validation purposes. You have quite a few options for doing so, such as:

Traverse HTML directly through the DOM
Using existing semantic HTML information
Adding semantic HTML information
Wrapping data with specific HTML

I’ll dismiss the first option immediately, it’s quite brittle to traverse only the DOM to locate data on a screen. If I have to have intimate knowledge of the entire HTML document rendered, the test will become far too dependent on the layout on the screen. If I decide to move information out of a table and into a definition list, my test will likely break.

From there, I’m left with basically two options – semantic HTML, whether it’s existing or added, or wrapping each data piece with custom HTML. In the presentation, two questions came up:

When locating a piece of information in a row in a table, why not use something more meaningful, such as the underlying entity’s ID to tag the data?
Again in the table, why not use semantic information on the row, instead of adding meaningless HTML?

Before we look at the different ways to verify content, let’s first look at what we want to verify.

The original content

In the scenario I highlighted in the screencast, I walk though editing some information, then verifying that data was changed correctly. I still have tests for controllers, repositories and so on, but I want a high level scenario-based test to make sure the entire pipeline is working as expected. We try and pull the information out of the HTML content rendered, using some sort of UI testing tool. Without modifying the content, here’s what we have to work with:

<div id="main">
    <h2>Products</h2>
    <table>
        <thead>
            <tr>
                <td>Details</td>
                <td>Name</td>
                <td>Manufacturer</td>
                <td>Price</td>
            </tr>
        </thead>
        <tbody>
            <tr>
                <td><a href="/Product/Edit/1">Edit</a></td>
                <td>Insignia® - 26" Class / 720p / 60Hz / LCD HDTV DVD Combo</td>
                <td>Insignia</td>
                <td>359.99</td>
            </tr>
            <tr>
                <td><a href="/Product/Edit/2">Edit</a></td>
                <td>Insignia® - 19" Class / 720p / 60Hz / LCD HDTV</td>
                <td>Insignia</td>
                <td>189.99</td>
            </tr>
            <tr>
                <td><a href="/Product/Edit/3">Edit</a></td>
                <td>Dynex® - 15" Class / 720p / 60Hz / LCD HDTV</td>
                <td>Dynex</td>
                <td>159.99</td>
            </tr>
        </tbody>
    </table>

There’s more rendered than just this HTML, but this is the piece that really matters. We have some semantic CSS information, the DIV with an ID of “main”, but that’s about it. In our CSS, we style our tables based on an ID-Element selector, but we could have just have easily used a class name on the table. I’d like to verify that the price on the first product was changed correctly, but how do I verify this? How do I make sure that the value in that one cell is the new price?

Choosing a strategy

I don’t want false positives or false negatives. I don’t want to look for just any table or any table cell in the markup, I want really the piece just around “359.99”. A few options to do so include:

Locating the outer DIV, finding the table, the first row, last cell, etc.
Apply a special class to the TD just where that price is shown
Add markup only around data-bound elements

When I first started doing UI testing, I assumed that I wanted to be as semantic and pure as possible, and not try to change my markup just for tests. That idea lasted about an hour, and reality set in that in general, things not designed with testing in mind will be hard to test. Hard to write tests, hard to maintain tests.

My next idea was to add information as needed, perhaps a class name to an existing surrounding element that might double for styling. But I’ve moved completely away from that strategy as well, as styling and UI testing are two different concerns that should not mix.

Why keep these concerns separate? Frankly, because both concerns have two completely different reasons to change. Styling and behavior (such as IDs added for jQuery) are about design and interaction. UI testing is about verifying site behavior. Mixing the two concerns means my tests are more likely to break because I’ve changed the design. But design can change without behavior changing, so I don’t want to couple my tests to the visual design. Coupling UI tests to the site design is like unit testing using only the reflection API.

Instead, I’d rather couple my tests to something that better represents the view – and that would be the ViewModel.

Strongly-typed views, strongly-typed tests

We use strongly-typed views because using a dictionary just leads to brittleness and fear of change. Strongly-typed views, along with the concept of the Thunderdome Principle, let us move past the issues of broken views and broken model binding. Instead of writing black-box tests against our UI, we can take advantage of the strongly-typed views, and modify our HTML rendered to get access to the rendered ViewModel. If you look at views from the point of view of “how to render a ViewModel”, then your UI tests can now talk in the language of the ViewModel.

But to help our UI tests out, we need to be able to locate information rendered from a ViewModel. Looking at our original HTML, the most fool-proof, straightforward way to do this would be to add HTML as close to the rendered ViewModel as possible. Let’s look at our view to see where this might be:

    <h2>Products</h2>
    <% var products = Model; %>
    
    <table>
        <thead>
            <tr>
                <td>Details</td>
                <td>Name</td>
                <td>Manufacturer</td>
                <td>Price</td>
            </tr>
        </thead>
        <tbody>
        <% foreach (var product in products) { %>
            <tr>
                <td><%= Html.ActionLink("Edit", "Edit", new { id = product.Id }) %></td>
                <td><%= product.Name %></td>
                <td><%= product.ManufacturerName %></td>
                <td><%= product.Price %></td>
            </tr>
        <% } %>
        </tbody>
    </table>

So what’s the ideal spot here? On the table? The table body? Row? Cell? These are all possible…and all coupled to the design of the UI, rather than the shape of the ViewModel. Instead of coupling to the design of the HTML, we can instead change every spot where we render the ViewModel directly to have extra HTML used only for UI tests. In the presentation, I move towards this model:

<tbody>
<% var i = 0; %>
<% foreach (var product in products) { %>
    <tr>
        <td><%= Html.ActionLink("Edit", "Edit", new { id = product.Id }) %></td>
        <td><%= Html.Span(m => m.Name) %></td>
        <td><%= Html.Span(m => m.ManufacturerName) %></td>
        <td><%= Html.Span(m => m.Price) %></td>
    </tr>
<%
    i++;
} %>
</tbody>

In practice, I’ll use a similar concept to the Opinionated Input Builders Eric highlighted a while back. Instead of input elements, I’ll solely render at the core, span tags. Here is the rendered HTML:

<div id="main">
    <h2>Products</h2>
    <table>
        <thead>
            <tr>
                <td>Details</td>
                <td>Name</td>
                <td>Manufacturer</td>
                <td>Price</td>

            </tr>
        </thead>
        <tbody>
            <tr>
                <td><a href="/Product/Edit/1">Edit</a></td>
                <td><span id='_0__Name'>Insignia® - 26" Class / 720p / 60Hz / LCD HDTV DVD Combo</span></td>
                <td><span id='_0__ManufacturerName'>Insignia</span></td>
                <td><span id='_0__Price'>359.99</span></td>
            </tr>
            <tr>
                <td><a href="/Product/Edit/2">Edit</a></td>
                <td><span id='_1__Name'>Insignia® - 19" Class / 720p / 60Hz / LCD HDTV</span></td>
                <td><span id='_1__ManufacturerName'>Insignia</span></td>
                <td><span id='_1__Price'>189.99</span></td>
            </tr>
            <tr>
                <td><a href="/Product/Edit/3">Edit</a></td>
                <td><span id='_2__Name'>Dynex® - 15" Class / 720p / 60Hz / LCD HDTV</span></td>
                <td><span id='_2__ManufacturerName'>Dynex</span></td>
                <td><span id='_2__Price'>159.99</span></td>
            </tr>
        </tbody>
    </table>

Notice that each data-bound output is wrapped with a span tag – and a special span tag at that. The span tag has a unique ID that represents the expression used to render that piece of data. The first product name shown comes from the expression “m => m[0].Name”. Because I know the ViewModel used to render this view, I can use an expression combined with a UI testing tool to locate the rendering of any piece of information on the screen.

The downside of this strategy is the amount of HTML added to the final document render. But since we’ve separated HTML used for styling and behavior, and HTML rendered strictly for testing, we can put a switch in our application so that for environments not doing UI testing, the span tag isn’t ever rendered! We can then get the best of both worlds – non-obtrusive JavaScript on top of semantic HTML, and flip a switch to be able to get to any piece of information on the rendered HTML.

Because our UI tests also use the ViewModel types and expressions to locate the same rendered ViewModels that the views used to render, we no longer run into the issue where our UI tests become disconnected from the views rendered. If we delete a property on our ViewModel, guess what, the UI test no longer compiles!

Wrapping it up

Developing a UI testing strategy isn’t the easiest thing in the world. In our current project, I think we had three or four evolutions before we finally settled on a UI testing strategy we like. And it’s still not perfect – we’re still looking at ways to make UI testing easy, expressive, and more valuable. But gone are the days where a styling change broke our tests. When our UI tests fail now, it’s more along the lines of a new required field, changing business rules and so on.

I don’t like mixing styling and behavior with locating data-bound UI elements, simply because both have very different reasons for change. But by taking advantage of our strongly-typed views in our UI tests, we can render the ViewModel in such a way that makes it dirt-simple to locate individual pieces of our ViewModel on the final rendered page.

C4MVC UI Testing screencast posted

bogardj — Thu, 22 Oct 2009 18:37:41 PDT

Yesterday the C4MVC guys hosted a screencast with me on UI testing ASP.NET MVC. It was a lot of fun, but I actually forgot I was being videotaped (webcam’d?), so all those times I turned off my mic to catch a drink of water or cough uncontrollably are caught in glorious 370i. Anyway, check out the video here:

Unfortunately, I didn’t leave enough time for questions at the end, and I had some good ones from the audience. You can also find the complete code and slides here.

Thanks again to the C4MVC guys for hosting me! Since I didn’t have time to answer too many questions, feel free to ask them in the comments, or you can always use the Contact Me link.

C4MVC meeting on UI testing

bogardj — Wed, 21 Oct 2009 06:38:41 PDT

Today I’m showing giving a talk on UI testing MVC applications for the Community for MVC.Net folks. Those that went to my presentations in Arkansas and Houston have seen this talk already, so they’re off the hook on this one. I’ll be covering ways to create UI tests that don’t break when a butterfly flaps its wings in Japan, and how we can design our MVC applications for UI testability.

The presentation is done via LiveMeeting, (http://Meeting.c4mvc.net), so make sure you’ve hit that link before the start time, which is 12:00 PM CST today. For more details, check out the C4MVC site:

http://www.c4mvc.net/

In the WebForms world, it was quite difficult to create maintainable UI tests. But in ASP.NET MVC, we have far more control over the HTML generated, allowing us to design our UI for testability. It should be a good discussion, I’m looking forward to seeing everyone there!

How not to implement a failing test

bogardj — Mon, 19 Oct 2009 06:45:30 PDT

One of the first things I change in ReSharper, along with one of my biggest pet peeves is a failing test that fails because of something like this:

public class CombinedStreetAddressResolver 
    : NullSafeValueResolver<Address, string>
{
    protected override string ResolveCore(Address model)
    {
        throw new NotImplementedException();
    }
}

In the Red-Green-Refactor progression, the Red of a failing test should come from an assertion failure not a “my code is stupid” failure. The Red part is intended to triangulate and calibrate your test, to make sure that your test can fail correctly. A NotImplementedException won’t cause a meaningful failure, and only serves the purpose of getting your code to compile.

Throwing exceptions means your assertions never get executed in RGR until you attempt to make a passing test, in which case you still haven’t proven your test to be correct. If you don’t know that your test is correct, you have two points of failure: your test, and code under test.

That’s why I’ve set ReSharper to return default values instead of throw exceptions. I want meaningful failures from a valid test, otherwise I’m better off skipping the Red step altogether.

More missing LINQ operators

bogardj — Thu, 15 Oct 2009 20:19:55 PDT

Continuing an old post on missing LINQ operators, the wonders of extension methods allow us as developers to fill potential holes in LINQ operators. Whether it’s a Zip method (now included in .NET 4.0), or better methods for IComparer-based operators, I find myself adding more and more helpful LINQ operators, I wish were already in the framework.

Alternate

Do you ever want to weave two collections together, like shuffling a deck of cards? Well I know I do! Suppose we have this collection:

[1, 3, 5]

And this collection:

[2, 4, 6]

I’d like to create new collection that is the alternating items from the first and second list:

[1, 2, 3, 4, 5, 6]

Here’s the code to do it:

public static IEnumerable<TSource> Alternate<TSource>(this IEnumerable<TSource> first, IEnumerable<TSource> second)
{
    using (IEnumerator<TSource> e1 = first.GetEnumerator())
    using (IEnumerator<TSource> e2 = second.GetEnumerator())
        while (e1.MoveNext() && e2.MoveNext())
        {
            yield return e1.Current;
            yield return e2.Current;
        }
}

Very simple, I iterate both enumerables at the same time, yielding the first, then second collection’s current item. So how is this useful? How about this action:

[Test]
public void Word_play()
{
    var source = new[] {"The", "quick", "brown", "fox"};

    var result = source.Alternate(Spaces()).Aggregate(string.Empty, (a, b) => a + b);

    result.ShouldEqual("The quick brown fox ");
}

private IEnumerable<string> Spaces()
{
    while (true)
        yield return " ";
}

I cheated a little bit with an infinite sequence (the Spaces() method), but I found this method useful when I had to split, then reconstruct new sequences of strings.

Append

I really hate this syntax:

[Test]
public void Bad_concat_method()
{
    var ints = new[] {1, 2, 3};

    var oneToFour = ints.Concat(Enumerable.Repeat(4, 1));

    CollectionAssert.AreEqual(new[] { 1, 2, 3, 4 }, oneToFour.ToArray());
}

I want to just stick an item on the end of an existing collection, but I have to use this arcane Enumerable.Repeat method to do so. Instead, let’s create an operator that lets us tack an item on to the end of a collection:

public static IEnumerable<TSource> Append<TSource>(this IEnumerable<TSource> source, TSource element)
{
    using (IEnumerator<TSource> e1 = source.GetEnumerator())
        while (e1.MoveNext())
            yield return e1.Current;

    yield return element;
}

Now our code becomes much easier to understand:

[Test]
public void Easier_concat_with_append()
{
    var ints = new[] {1, 2, 3};

    var oneToFour = ints.Append(4);

    CollectionAssert.AreEqual(new[] { 1, 2, 3, 4 }, oneToFour.ToArray());
}

Prepend

Append wouldn’t be complete without the converse, Prepend, now would it?

public static IEnumerable<TSource> Prepend<TSource>(this IEnumerable<TSource> source, TSource element)
{
    yield return element;

    using (IEnumerator<TSource> e1 = source.GetEnumerator())
        while (e1.MoveNext())
            yield return e1.Current;
}

Now putting something on the beginning of a list is easier as well:

[Test]
public void Easier_concat_with_prepend()
{
    var ints = new[] {1, 2, 3};

    var zeroToThree = ints.Prepend(0);

    CollectionAssert.AreEqual(new[] { 0, 1, 2, 3 }, zeroToThree.ToArray());
}

Much more readable. I have a few more around replacing the IEqualityComparer<T> overloads, but those are a little bit more esoteric in their examples of crazy set-based logic. What’s really cool about all these methods is they still allow all sorts of fun chaining, allowing me to create very terse chains of operations on lists, with what would have taken a bazillion cryptic for..each loops. Cool stuff!

Thanks Arkansas!

bogardj — Thu, 15 Oct 2009 19:01:17 PDT

This week I made the drive from Austin to Fort Smith and Springdale, Arkansas to speak at the Fort Smith DNUG and the Northwest Arkansas DNUG on UI testing ASP.NET MVC. Although I showed tools such as WatiN and Gallio, I focused more on design for UI testability, and covered items such as:

Site navigation
Filling out forms
Verifying results

We stepped through a working, but horribly fragile UI test, and walked through how we can put in a few design techniques to share knowledge between view design and the UI test. Hopefully, my message of “ban test recorders” gets through to someone, as the UI test recorder is something that threw me off for years on creating maintainable UI tests.

You can find the slides and code here:

Testing the last mile in ASP.NET MVC

Thanks to all the guys in Arkansas for the great conversations, and I hope to see everyone soon! One parting shot from a trip to Devil’s Den:

My jeans were soaked up to my knees from that hike…

An AutoMapper success story

bogardj — Thu, 08 Oct 2009 09:38:32 PDT

I got a cool message on the AutoMapper mailing list from Howard Van Rooijen on how they used AutoMapper in a site they recently launched to production:

Hello AutoMapper Community,

I just wanted to let you know that an e-commerce site which uses AutoMapper as part of it's core architecture has just been released into the wild:

http://www.fancydressout fitters.co.uk

And I wanted to say a HUGE thank-you to Jimmy & the Community for this wonderful tool - that helps remove so much commodity plumbing code from the solution.

We were a little sceptical at the start of the project that AutoMapper would "cut the mustard" when it came to the performance requirements of a public facing, high load, e-commerce site because of the amount of reflection AutoMapper uses at its core, but we have been incredibly impressed with the performance of the solution under load.

The site is based on the S#arp Architecture Framework and its become very apparent how well AutoMapper fits into MVC style architecture as it enables easy separation of concerns with regards to object conversion (entities & ViewModels). Once we moved from hand-cranked converters to AutoMapper it was amazing how much cleaner our code became - so much so that we modified the overall solution architecture to incorporate explicit mapping layers (see attached image). Our general pattern of usage within MVC is as follows:

1. Map input into Domain Entities in the Controller

2. Pass Domain Entities into Task Layer to "do stuff"

3. Map output of Task Layer (Domain Entities) into ViewModel

4. Pass ViewModel to ViewEngine

Simple, slick and clean.

To formalise the Mapping Layer and make it testable we implemented a simple interface:

public interface IMapper<TInput, TOutput>

{

TOutput MapFrom(TInput input);

}

Next we'd implement a custom marker interface so that we could resolve the mapper from the DI container and we adopted the naming convention <Input Type><Output Type>Mapper:

public interface IEditModelEntityMapper : IMapper<EditModel, Entity>

{

}

Then finally implement the interface:

public class EditModelEntityMapper : IEditModelEntityMapper

{

public EditModelEntityMapper()

{

Mapper.CreateMap<EditModel, Entity>()

.ForMember(x => x.Property, y => y.MapFrom(z => z.Property));

}

public Entity MapFrom(EditModel input)

{

return Mapper.Map<EditModel, Entity>(input);

}

}

Then to actually use it in the MVC app:

public class CustomController : Controller

{

private readonly ITasks tasks;

private readonly IEditModelEntityMapper editModelEntityMapper;

private readonly IOutputViewModelMapper outputViewModelMapper;

public CustomController(ITasks tasks,

                                           IEditModelEntityMapper editModelEntityMapper,

                                           IOutputViewModelMapper outputViewModelMapper)

{

this.tasks = tasks;

this.editModelEntityMapper = editModelEntityMapper;

this.outputViewModelMapper = outputViewModelMapper;

}

public ActionResult Index(EditEntity input)

{

var entity = this.editModelEntityMapper.MapFrom(input);

                var output = this.tasks.DoSomething(entity)

                var viewModel = this.outputViewModelMapper.MapFrom(output);

return View(viewModel);

}

}

For those who are interested - here's a little more info:

- We ran the project using Scrum and delivered in 20 weeks: 10 x 2 week iterations

- It's based on the S#arp Architecture framework, which we extended to support, AutoMapper, Spark and ViewModels
- Solution performs very well: 1000 concurrent users per web server, generating around 180 pages per second across 2x single quad core 64bit servers.

Again, many thanks,

Howard

Howard also shared a neat little diagram of his architecture:

This is one of the greatest feelings from doing OSS – that something you created basically just for yourself can also help out other folks out there trying to deliver value for their customers. Thanks for all the feedback everyone, as well as kudos to the S#arp Architecture team for building such a great framework for MVC.

Bugs, defects and feedback

bogardj — Wed, 30 Sep 2009 10:58:35 PDT

In my last post, I talked about how we like to track bugs, with just pieces of paper. At lot of the responses were interesting, but I think some came back to the issue of “what is a bug?” Some felt that bugs needed to live in a more durable system (as if paper isn’t durable, and previous to around 1985 no one knew how to manage information). But for us, bugs don’t live more than 1 business day, so there’s no need to keep them around, unless we want to do trend analysis.

But one important issue is how we deal with feedback. A user can come up to us and say, “the site is broken”. That could mean a lot of things. It could mean the site is blowing up, or it could mean that they couldn’t find the “Submit Order” button. To the user, the site is broken.

Instead of referring to negative user feedback as “bugs”, we treat them as “issues”. So what is an “issue”? An issue is negative user feedback with potentially actionable fix. Note “potentially”.

The feedback process

When we get an issue from a user, the issue goes into triage with the Product Owner. The Product Owner is the person/team deciding what gets developed, so it’s logical that they decide what action to take with an issue. An issue falls into two broad categories:

Bug
Defect

A bug is an issue where the site is unusable or acceptance criteria has not been met. This includes things from “the site is down” to “you misspelled a word”. All of our stories have tangible and intangible acceptance criteria, functional and non-functional requirements. The expectations of these have been laid forth a long time ago, so we don’t really negotiate too much on what is or is not a bug. The user may have a different idea of the classification of bug, but they don’t get to decide.

A defect is an issue where the site is usable and acceptance criteria have been met, but the application does not work as expected. Things in this category are “data field should be over here”, “add this label”, “this business rule isn’t quite right” and so on. We don’t expect the users to know all of the stories we’ve developed, but there are still situations where the site isn’t working like it should, but works as agreed. We work hard to do good JIT analysis, but we still miss things.

Actions on bugs

Bugs are fixed immediately, with an SLA of one business day. We do things like branch and release-per-feature, automated builds and deployments, so we’re never more than about 15 minutes away from a commit to production fix. We refuse to keep an inventory of open bugs, so bugs are fixed as they arrive, and the team will stop their work to fix a bug. Because we stop-the-line, we strive to eliminate as many bugs as we can. This includes good design, a strong testing culture, and post-mortems on bugs to see how we can improve the next time around and eliminate the possibility of a bug like that to arise again.

Actions on defects

Defects need to be analyzed just as much as regular stories need to, so defects enter a second queue where they either turn into stories, or something we call “tweak stories”. Stories take 1-5 business days to deliver, while a “tweak story” gets turned around as quickly as a bug. It all comes down to the complexity of the defect. In any case, the Product Owner prioritizes these defects along with stories, and ultimately it’s the PO that decides when (or if) the team takes action on a defect.

The feedback cycle

For critical bugs, like “the site blows up”, there is no need to triage. It’s unacceptable that the site blows up, so we’ll fix those immediately. But for more subtle feedback, the developer team is ill-equipped to make a determination on the nature of an issue. Often, the development team attempts to fix every issue that comes their way, whether it’s a bug or not, in an effort to make the user happy. But the guy that says “the customer is always right”? He goes broke. The customer isn’t always right, so it’s the PO’s responsibility for determining the classification of the issue.

Ultimately, it’s about roles and responsibilities. Developer teams are usually quite poor at determining the priority of work, as they don’t always have the deep business/project understanding of scope, deadlines, priorities and so on. Instead, we place the responsibility of triage and prioritization where it belongs, with the Product Owner.

With this system in place, we have no need to keep what we call “bugs” around in a durable format. Once artifacts become durable, people hesitate to delete them, and now they need to become managed. At one point, I kept bugs. But I found I never looked at them, as our practice of post-mortems on bugs as they are fixed was perfectly adequate. Instead, I (or nowadays, the originator) just toss them when they’re done, and it feels quite liberating.

My favorite bug-tracking system

bogardj — Tue, 29 Sep 2009 10:57:03 PDT

I’m of the opinion that a process must demonstrate the need for software, before software is put in place to manage that process. Bug tracking is a process, but often we jump straight to a software solution for managing/tracking bugs before considering lower-tech, lower-risk and lower-overhead approaches. In our bug-tracking process, we employ a few main principles:

Bugs are a normal part of the development process
Most bugs are rework, therefore waste, and we should strive to eliminate this waste
Bugs are fixed as they are found (stop-the-line), and do not live for more than 1 business day

Given that bugs do not live for more than one business day, it is not necessary for us to maintain an inventory of bugs. Instead, we want a system than can accurately report a bug, and make it easy for it to be verified.

With these constraints in mind, my favorite bug tracking system is a 5x8” orange sticky note:

On this giant sticky note, the person that finds the bug writes:

Their initials
The name of the story
A short description

If there is a screenshot, this will be stapled to the back. When the developer is done with the bug, they write:

Their initials
The revision # of the commit that fixed the bug

The revision number is used to make sure that the currently deployed version is up-to-date enough to check the fix. So how does this bug tracking system work? Here is our process:

Person finds bug. Person could be anyone, developer, BA, product owner.
Finder creates orange card for bug, following the specified template (every piece of information goes in a certain spot on the giant post-it)
If the bug is a blow-up, or blocks the delivery of a story, it is placed on the desk of the tech lead (me)
If the bug is not specifically called out in acceptance criteria, it is placed on the desk of the BA lead.

These bugs go through triage, because sometimes things “work as designed” but not “work as expected”. Those in the latter category need to go through the normal story process.

The tech lead assigns the bug to a developer (or themselves), by placing the bug on the assigned developer’s desk
When the fix is done, the developer writes their initials and the revision number, and gives the orange card back to the originator, with a smile
The originator of the bug verifies the fix, and places the orange card in the trash with a “thumbs up” to let the developer know the bug is verified fixed.

This is about as lightweight as we can get it, without chaos. This process has evolved over time, but we’ve always fit our tools (post-its) to our process, instead of fitting our process to a tool. We’re all in the same room, so there is absolutely zero need to track bugs electronically. Tracking them electronically would create waste, as we would now have inventory to manage in some other system. It really can’t get much easier than a bright piece of paper on my desk.

Fit the tool to the process

This process evolved given the attributes of the constraints of our system. We all sit in the same room. We strive for face-to-face, verbal communication over all other forms. We don’t want to add overhead by tracking information that no one cares about. When someone suggests to use a software bug tracker, it needs to come from a need to have that information in that format, not because geeks like software.

But what if we had different constraints? What if we weren’t all in the same room? Fine, software it is! Until then, I’ll choose the absolute lowest overhead solution.

But how does it scale? Well, if you keep an inventory of bugs for more than a day, not too well. If bugs are fixed when they happen, it’s just not possible to have more than a dozen bugs out at a time. Scaling needs a vector, you have to understand in which direction it can scale. It scales with the size of the team, but not with geography or volume. That’s fine, we keep those two small and constant. Those choices were intentional, as separated teams and higher volume through poor quality or turnaround decreases delivery throughput.

Thanks Houston TechFest!

bogardj — Mon, 28 Sep 2009 05:41:01 PDT

This past weekend I gave a talk on one of my favorite topics – UI testing. In it, I focused almost exclusively on techniques for authoring maintainable UI tests, and how design for testability extends to views, models and controllers in MVC land. It’s trivial to write UI tests that break easily, but not much more difficult to write maintainable tests, given that you’ve intentionally designed your UI layer for UI testability.

Big thanks to the Houston TechFest organizers, who put on a fantastic conference, and to all the attendees, who had to put up with me for an hour. For those that are interested, you can find the code and slides here:

Testing the Last Mile in ASP.NET MVC code and slides

I had a lot of people asking about the wrappers on top of our browser automation tool, WatiN, and if they were open source. The answer is no, not yet, but we’re working on it. A quick road to fragile UI tests is browser automation code directly in your UI tests, so the sample code provided should at least provide some ideas and guidance to craft your own wrappers. And as always, my door is always open for questions. My email door, at least.

The case for two-way mapping in AutoMapper

bogardj — Thu, 17 Sep 2009 18:33:17 PDT

I’m getting more and more requests around the area of two-way mapping, meaning you’d do something like:

Product –> ProductDTO

ProductDTO –> Product

Product being an entity, I can’t for the life of me understand why I’d want to dump a DTO straight back in to a model object. To get some understanding of how we use AutoMapper, we have:

1500 individual mapping definitions
75 custom value resolvers
38 custom value formatters
5 individual profiles

So what are we using AutoMapper for? Our five profiles include:

From Domain to ViewModel (strongly-typed view models for MVC)
From Domain to EditModel (strongly-typed view models for forms in MVC)
From EditModel to CommandMessages – going from the loosely-typed EditModel to strongly-typed, broken out messages. A single EditModel might generate a half-dozen messages.
From Domain to ReportModel – strongly-typed Telerik reports
From Domain to EDI Model – flattened models used to generate EDI reports

There is no two-way mapping because we never need two-way mapping. There was a point very early on where we were at a critical junction, and could decide to do two-way mapping. But we didn’t. Why? Because then our mapping layer would influence our domain model. I strongly believe in POCOs, and a very writeable domain model meant that POCOs were out. What exactly would two-way mapping do to our domain layer?

Force mutable, public collection , like “public EntitySet<Category> Categories { get; }” <- NO.
Make testing much, much harder, as we only ever wanted to update a portion of a domain model
Force our domain model to be mutable everywhere

So my question to those wanting two-way mapping:

What scenarios are you looking at doing two-way mapping?
What impact would two-way mapping have on the originating source type?
How would you test two-way mappings?

I think using AutoMapper because you don’t want to use the “=” operator is a bit lazy. Instead, we use it to flatten and reshape, optimizing for the destination type’s environment. Remember, my original motivation for AutoMapper was:

Enable protecting the domain layer from other layers by mapping to DTOs
Enable easy testing of the mappings, which would otherwise prevent us from creating the mapping in the first place

So…why two-way mapping?

AutoMapper 1.0 RC1 released

bogardj — Mon, 14 Sep 2009 17:34:23 PDT

It’s been quite a long journey with AutoMapper, with the origins written just over a year ago now. I’ve focused on stability and performance since the 0.3.1 release back in May, and from here to the 1.0 release, I’ll just be doing bug fixes. I did work in quite a few new enhancements, but I’m waiting on bigger changes until after the 1.0 release. From the CodePlex download site:

New Features

Added non-generic CreateMap overload
Can specify custom mapping ordering for individual destination members
Before/After map callbacks for custom pre/post processing
Registration of custom pre- and postfixes on member type names (i.e. CustomerKey can map to Customer)
Mapping from dictionaries to split out key-value pairs
Basic support for IDataReader/IDataRecord
Support for custom naming conventions
Support for IListSource (for Entity Framework)

Enhancements

AllowNullDestinationValues now defaults to "true"
Mapping operations thread-safe
Performance improvements for mapping pipeline, including late-bound expression trees and Lightweight Code Generation
Allowing using the destination value for individual members
Lots of internal refactoring around the mapping engine to support various IoC scenarios
Assembly marked as CLS compliant
Registering global and profile-specific aliases for names
Support for custom destination type constructors
Upgraded to latest LinFu release

Bug Fixes

Validation errors on explicitly implemented interfaces
Collections with null elements caused exceptions
Readonly destination members were causing validation errors
Removed null checking in custom value resolvers

Enjoy!!!

Wither the Repository

bogardj — Thu, 10 Sep 2009 19:05:00 PDT

Looking at the different Repository pattern implementations, one thing really surprised me – how far off these implementations are from the original Fowler definition of the Repository. Instead, we see a transformation to the examples in the Evans description of Repository. Fowler’s definition for a Repository is:

Mediates between the domain and data mapping layers using a collection-like interface for accessing domain objects.

But in our original Repository pattern implementations, we see far more than Querying, but persistence as well (Create, Update and Delete). This is consistent with some of the descriptions in Evans book, where the collection-like interface also allows for modifications – adding and removing – thus completing the lifecycle of a domain object.

But is this Repository pattern needed? What exactly is the Repository buying us? In normal usage, I see the Repository pattern doing two main tasks:

Encapsulate the data mapping layer
Encapsulating difficult queries (in the Named Query Method approach)

In some systems, including the one I’m working on right now, each entity corresponds 1 to 1 with a repository implementation – whether or not the repository implementation is doing anything. Looking back at the Fowler PoEAA book, there is another interesting pattern at play – the Unit of Work pattern. The Unit of Work:

Maintains a list of objects affected by a business transaction and coordinates the writing out of changes and the resolution of concurrency problems.

A common Unit of Work implementation wraps the underlying data access layer, and provides means of creating transactional boundaries:

public interface IUnitOfWork : IDisposable
{
    void Initialize();
    void Commit();
    void Rollback();
}

public interface INHibernateUnitOfWork : IUnitOfWork
{
    ISession CurrentSession { get; }
}

The interesting piece here is the ISession interface from NHibernate. It puts several key patterns together – Unit of Work, Identity Map, as well as the Data Mapper pattern. Now, I’ve never been on a project that has wanted or needed to swap out ORM solutions (provided they chose an active, mature solution), so what is this encapsulation of the Repository buying us exactly?

A bigger issue comes into play when trying to implement Command-Query Separation throughout your application – the Repository pattern just does not support it well enough. Eventually, you’ll likely be left with a class that is Repository in name only, but not the original intent. Let’s look at some of the problems of Repository to get an idea how CQS doesn’t gel well.

Loose aggregate boundaries

The problems Aggregate Roots solve is one of spaghetti-coded associations between entities, where I could never figure out the right place to draw the boundary. I originally assumed that a save operation meant that you would save a single aggregate root, and cascading would take care of the rest. Unfortunately, that simplicity and rigidity tends to fall flat in larger models. In an aggregate root, child entities only have identity within the boundaries of the root, and no outside identity.

In the web world, that would mean that anything with an Edit link is inherently an aggregate root, as it needs its own global identity to save properly. We toyed around with local identity, and always passing around the parent object to work on the child. But all that extra work didn’t buy us anything. Local identity only bought us anything when you edited an entire entity on one page, including children.

Cascading poses other issues. What we found is that cascading depends not on the entity I’m working with, but the specific operation/command I’m trying to perform. Instead, we’ve seen aggregate boundaries depend completely on the command we perform. Sometimes the root is object A, sometimes it’s B. Instead of worrying too much about aggregate boundaries, we greatly reduced the relative boundaries of roots, until it was a use-case per use-case basis that determined where we drew our boundaries.

Because roots can hold references to other roots, the concept of a boundary and local identity eventually restricted itself to basically screens where you saved the parent and children in one operation. Again, it was the C in CQS that drove boundaries.

Thus it started to become confusing how we should handle cascades and save logic. Since boundaries varied screen-to-screen, it became obvious that any custom save logic in a Repository could be handled perfectly well in the ORM layer, with cascades. Eventually, our model became extremely connected, with appropriate cascading and modification operations available only when specifically needed.

First-class queries

On virtually every screen our application, we show the exact same entity. That’s around 150 screens or so, where what you see begins with one Person, and filters down from there. This made things quite interesting from the query optimization standpoint. We had two choices – go through the root Person entity to all of its children, in one slice. Or, go from the children, and get back to the parent. But sometimes we show bits and pieces of various slices of our Person object, things that did not fit well with just one Repository call. We enabled lazy loading – but only because we wanted a well-connected object model, with the knowledge that we would optimize the access later.

When we actually got to the optimizations, we could either create a bunch of GetByIdForAbcScreen on our PersonRepository, or, encapsulate all of the fetching in a single Query object. Guess which route we went…

With first-class query objects in our system, the need for a true PersonRepository becomes much less. It doesn’t do any custom queries, those are now in distinct query objects, designed for each situation.

Wither the Repository?

No, not yet. Complexity in a system is never uniform, and a single solution rarely fits every circumstance. The tough part is figuring where to put custom persistence or fetching logic. New query method on a Repository? Override the Save method? Is that custom save logic on every save for this entity, or just for certain commands? Complexity resembles more the picture of cosmic background radiation from the Big Bang, almost random in nature, but a predictable and expected variation. Most spots are in the middle of the complexity scale, some spots are so easy that the architecture chosen may seem too much, but other spots that bend our architecture to its limits. The Repository is one of those spots.

Personally, I see a few trends in Repositories:

Trend towards a Generic Method Repository, where there is no custom logic for CRUD
Queries externalized and encapsulated in Query objects when needed, otherwise exposed through LINQ
Custom persistence logic externalized and encapsulated in Commands, that utilize the one Repository implementation as needed
Repository becomes relegated, in NHibernate terms, as the ISessionFacade

I’m not fully ready to chuck our custom repositories, as I’m still not quite fully cognizant of what this change might mean. But the parallel inheritance hierarchies are naive and a little difficult to work with, and don’t really reflect the reality we’re trying to model in our controller level of our application – modeling Queries and Commands as first-class citizens.

DDD: Repository Implementation Patterns

bogardj — Wed, 02 Sep 2009 18:00:32 PDT

One of the major structural patterns encountered in DDD (and one of the most argued about) is the Repository pattern. You’ve created a persistent domain model, and now you need to be able to retrieve these objects from an encapsulated store. In Fowler’s PoEAA, the Repository pattern is described as:

Mediates between the domain and data mapping layers using a collection-like interface for accessing domain objects.

In many DDD implementations, the use of a Repository is widened to go beyond retrieval, but the other pieces in the CRUD acronym. This is natural, as a Repository provides a centralized facade over some backing store, whether that backing store is a database, XML, SOAP, REST and so on. But what about the actual interface of the Repository? What kinds of options do we have for designing the actual Repository itself? There are quite a few different schools of thought here, each with their own benefits and drawbacks.

The Generic Repository Interface

In the generic repository interface, the interface definition itself is generic, with implementations for individual entity types. The generic type parameter denotes the entity type of the repository. One example is in S#arp Architecture:

public interface IRepositoryWithTypedId<T, IdT>
{
    T Get(IdT id);

    IList<T> GetAll();

    IList<T> FindAll(IDictionary<string, object> propertyValuePairs);

    T FindOne(IDictionary<string, object> propertyValuePairs);

    T SaveOrUpdate(T entity);

    void Delete(T entity);

    IDbContext DbContext { get; }
}

If you want a repository for a specific type of entity, you create a derived type, closing the interface type with the entity:

public interface ICustomerRepository : INHibernateRepositoryWithTypedId<Customer, string>
{
    List<Customer> FindByCountry(string countryName);
}

Note that this repository supplies an additional, specialized query method for a specific find method. The nice thing about a generic repository interface is that a corresponding base repository implementation can be designed so that all of the data access code that doesn’t change from repository to repository can have a place to live. The actual implementation can then rely on a common base implementation for the non-changing behavior:

public class CustomerRepository : NHibernateRepositoryWithTypedId<Customer, string>, ICustomerRepository
{
    public List<Customer> FindByCountry(string countryName) {
        ICriteria criteria = Session.CreateCriteria(typeof(Customer))
            .Add(Expression.Eq("Country", countryName));

        return criteria.List<Customer>() as List<Customer>;
    }
}

But, we still get the type safety of generics. However, we tend to muddy the contract of the ICustomerRepository a little if our entity does not support all of the operations of the base repository interface. It’s basically a sacrifice of a common base implementation that you violate the Interface Segregation Principle for some simplification on the implementation side. For example, in my current domain, deleting an entity has legal ramifications, so we basically don’t allow it for quite a few entities. Instead, things are marked as various stages of “soft” deletes.

One big benefit is that most modern IoC containers allow configuring the container such that the most derived type of IRepository<T> can be resolved. I can ask the container for IRepository<Customer>, and the CustomerRepository implementation is returned. This is especially helpful in APIs where we might abstract “saving” to some base layer. Again, the downside is that our interface has methods to support every operation, whether we actually want to support them or not.

The Generic Method Repository

In the generic method repository, our repository implementations do not expose methods for specific queries. Additionally, no specific repository for an entity is defined. Instead, only general-purpose methods are exposed for querying, persisting and retrieval. The methods are generic, providing type safety, but there is only one implementation. Consider the Alt.Oxite repository implementation from FubuMVC Contrib:

public interface IRepository
{
    void Save<ENTITY>(ENTITY entity)
        where ENTITY : DomainEntity;

    ENTITY Load<ENTITY>(Guid id)
        where ENTITY : DomainEntity;

    IQueryable<ENTITY> Query<ENTITY>()
        where ENTITY : DomainEntity;

    IQueryable<ENTITY> Query<ENTITY>(IDomainQuery<ENTITY> whereQuery)
        where ENTITY : DomainEntity;
}

In this implementation, the repository itself is not generic. A single repository dependency can perform all needed CRU (not D) operations against any well-known entity. Because most ORMs nowadays are generic, the implementation of a generic repository is very straightforward, it simply wraps the ORM. The actual definition of queries is left to something else. In the above case, LINQ queries utilize the Query method to retrieve persistent objects.

The advantage to this pattern is that we do not need to create an interface for each type of entity we create. Instead, we remove the responsibility of forming specific queries to the callers, including worrying about fetching, caching, projection and so on. Since this logic does not change, we don’t have to pull in a repository for every entity we want to retrieve. Instead, operations flow through one general-purpose interface.

The disadvantage to this pattern is we lose named query methods on our repository, as Greg pointed out a while back. The ICustomerRepository from the previous pattern exposed (and encapsulated) a well-known query with a intention-revealing name. With a generic method repository, there is only one implementation. Creating a derived ICustomerRepository of a generic method repository would be rather strange, as the base interface allows any entity under the sun.

The Encapsulated Role-Specific Repository

Classes that use a repository rarely use every method inside of it. A class that does a query against Customers probably isn’t going to then go delete them. Instead, we’re likely querying for some other purpose, whether it’s to display information, look up information for business rules, and so on. Instead of a catch-all repository that exposes every method under the sun, we could alternatively apply the Interface Segregation Principle for role-based interfaces, and define interfaces that expose only what one class needs. With these repositories, you’ll much more likely find encapsulated query methods, that do not expose the ability to arbitrarily query a backing data store:

public interface IProductRepositoryForNewOrder
{
    Product[] FindDiscontinuedProducts();
}

A single repository implementation implements all Product repository interfaces, but only the single method needed is exposed and used by the caller. This way, you do not see Save, Delete, and other query operations that you do not care about. The down side is that this implementation tends to be more difficult to combine with IoC containers, as auto-wiring does not work as expected. At some point, it starts to look like these types of repositories are glorified query objects, a single class that represents a query.

Wrapping it up

In our projects, we’ve tended to use the generic repository interface, as it allows us to override implementations for custom behavior. We still will include a default implementation that gets wired in, a non-abstract RepositoryBase<T>. The IoC container is configured to pick the most specific implementation, but if one doesn’t exist, we don’t need to create a blank repository. Sometimes, things like Save or Delete need to do custom things, and I’d rather put this in the repository rather than some other anonymous domain service.

In the end, it really depends on the situation of each project. I’ve tended to stick with a generic interface, as it’s the easiest to scale complexity. However, quite a few smart folks use the generic method repository, and rely on external queries for the hard stuff. Complexity with saves and deletes is then handled by the cascading behavior of the underlying ORM. It’s quite intriguing, but I’ve been reticent to give up a tried-and-true pattern. I’m curious to see what other repository patterns people use out there.