Jimmy Bogard

How not to do Dependency Injection, in NerdDinner

bogardj — Fri, 03 Jul 2009 09:19:29 PDT

Checking out the NerdDinner code the other day, I found a common Dependency Injection anti-pattern. One of the core concepts of DI is that components are not responsible for locating their own dependencies. The code went part of the way to full-on DI, but not quite far enough. Here’s the offending code:

public class SearchController : Controller {

IDinnerRepository dinnerRepository;

//
// Dependency Injection enabled constructors

public SearchController()
    : this(new DinnerRepository()) {
}

public SearchController(IDinnerRepository repository) {
    dinnerRepository = repository;
}

The second constructor is correct – the SearchController’s dependencies are passed in through the constructor. This is because the IDinnerRepository is a required, or primal dependency. SearchController depends on IDinnerRepository to function properly, and won’t work without it. But on the first constructor, we violate DI by having the SearchController create a concrete DinnerRepository! We’re now back to concrete, opaque dependencies. We have a small benefit of easier testability, but we still force our controller to locate its own dependencies.

So why is this a Bad Thing?

For one, it’s confusing to have two constructors. Why go through all the trouble of creating the IDinnerRepository interface if I’m just going to depend directly on an implementer? Now what if DinnerRepository now needs some dependency? What if it now needs some ILogger, security or policy dependency? Do I now need to go fix all of my calls to “new”?

And that’s the whole point of Dependency Injection, and the Dependency Inversion Principle. I only know about my first level dependencies, and abstractions on top of that. If we check out DinnerRepository, we see another issue:

public class DinnerRepository : NerdDinner.Models.IDinnerRepository {

    NerdDinnerDataContext db = new NerdDinnerDataContext();

A private, opaque dependency to NerdDinnerDataContext. If we wanted to make that opaque dependency explicit, we’d have to fix our SearchController (and all the other controllers as well). It’s these kinds of ripple effects that prevent refactoring and improvement.

Fixing it

But we can fix this, quite easily. From MvcContrib, we can grab any one of the ControllerFactory implementations for the IoC container of our choice. For me, that’s StructureMap. First, we’ll need to configure StructureMap to wire up all of our dependencies. The preferred way to do so is to create a Registry:

public class NerdDinnerRegistry : Registry
{
    public NerdDinnerRegistry()
    {
        Scan(scanner =>
        {
            scanner.TheCallingAssembly();
            scanner.WithDefaultConventions();
        });

        ForRequestedType<NerdDinnerDataContext>()
            .CacheBy(InstanceScope.HttpContext)
            .TheDefault.Is.ConstructedBy(() => new NerdDinnerDataContext());
    }
}

In the first part, we just tell StructureMap to scan the calling assembly with default conventions. This will wire up IDinnerRepository to DinnerRepository. Probably 99% of our dependencies will be taken care of in that one call. Next, we need to wire up the NerdDinnerDataContext (for reasons we’ll see soon). Since that class has multiple constructors, StructureMap likes to use the greediest dependency, with the most arguments. I don’t want that, so I override it to use the no-arg constructor. Finally, I cache it by HttpContext, though I could probably go Singleton if it’s expensive to instantiate.

Next, I need a wrapper class to initialize StructureMap:

public static class IoCContainer
{
    public static void Configure()
    {
        ObjectFactory.Initialize(init =>
        {
            init.AddRegistry<NerdDinnerRegistry>();
        });
    }
}

Initialize only needs to get called once per AppDomain, and all I need to do is add the NerdDinnerRegistry I created earlier. I could wire up everything here, but again, the preferred configuration method is through registries. The next piece is to wire up ASP.NET MVC to both call our configuration method and use the StructureMapControllerFactory:

void Application_Start()
{
    RegisterRoutes(RouteTable.Routes);

    IoCContainer.Configure();
    ControllerBuilder.Current.SetControllerFactory(new StructureMapControllerFactory());

    ViewEngines.Engines.Clear();
    ViewEngines.Engines.Add(new MobileCapableWebFormViewEngine());
}

I put all this in with the other MVC configuration in the Global.asax Application_Start method. One final piece is to remove the extra constructors on our controller:

public class DinnersController : Controller {

    IDinnerRepository dinnerRepository;

    //
    // Dependency Injection enabled constructors

    public DinnersController(IDinnerRepository repository) {
        dinnerRepository = repository;
    }

And make our DinnerRepository expose its dependencies explicitly:

public class DinnerRepository : NerdDinner.Models.IDinnerRepository {
    
    private readonly NerdDinnerDataContext db;

    public DinnerRepository(NerdDinnerDataContext db)
    {
        this.db = db;
    }

Note that there are no no-arg constructors to be found! When StructureMap creates a DinnersController, it will look to resolve the IDinnerRepository dependency. That’s a DinnerRepository, which in turn needs the NerdDinnerDataContext. But that’s all hidden from us, we never need to wire up an entire graph, as we would if we stuck to the “new” operator. Just to make sure everything works, I can navigate to view upcoming dinners:

Wrapping it up

In the original NerdDinner code, dependency inversion was only partially implemented as the original controllers still contained a constructor that called a constructor of a concrete class. To fix this, we added StructureMap to the mix, configuring ASP.NET MVC to use StructureMap to create controllers instead of the default controller factory. Finally, we pushed the dependency inversion principle all the way down to our repository, removing the opaque dependencies where we found them. When we finished, all of our classes exposed their dependencies explicitly through their constructor. No one class knew more than one level of depth, and our controllers now properly depended exclusively on the IDinnerRepository abstraction.

In the future, we can add things like logging, policies and the like to custom IDinnerRepository implementations, without needing to change any of our controllers. Once we introduce inversion of control to our application, we open a lot of doors for functionality and simplicity, but only going halfway won’t give us the full benefit.

Congrats to the Los Techies MVPs!

bogardj — Wed, 01 Jul 2009 18:03:38 PDT

This morning, I and fellow Los Techies alumni Eric Hexter and Chris Patterson learned that we are MVPs! Well, technically I learned last night through an IM of a re-tweet of my MVP lead, but hey, close enough, right? I know it’s cheesy, but I think these awards are a testament to the great community Los Techies has built over the years, so for me, this is really an award for Los Techies. The roundup includes:

Jimmy Bogard, MVP ASP.NET
Eric Hexter, MVP ASP.NET
Chris Patterson, MVP C#

Personally, I’ve viewed my community involvement as simply a desire to share my journey with others, and in turn learn where others have been and are going. My favorite experiences thus far have been the open space conferences, book clubs, dojos and code camps, as it gives me a great chance to meet and talk with a lot of smart people. Thanks again everyone!

How we do MVC – View models

bogardj — Mon, 29 Jun 2009 20:06:36 PDT

A while back, I went over a few of the patterns and opinions we’ve gravitated towards on our current large-ish ASP.NET MVC project, or, how we do MVC. Many of these opinions were forged the hard way, by doing the wrong thing many times until we found the “right” opinion. Of course, many of these opinions are only really valid in the constraints of our project. While the domain of this project isn’t important, here are some key aspects to consider:

AJAX is used very, very sparingly. Section 508 compliance is required
XHTML compliance is also required
XHTML DTD validation is also required
All (well, 99%) operations revolve a single uber-entity. Think customer relationship management, where everything you do deals with exactly one customer
Snippets of information repeated across many screens
Screens are either edit, or view, but never both. (99% never)

Given these constraints, these opinions may or may not apply to the project you work on. Again, patterns are all about tradeoffs, benefits and liabilities. But, opinionated software is like building a bullet train. It goes extremely fast, but only in the direction you build it.

That said, I’m going to go over some of the main aspects of our MVC usage in a series of posts – starting with ViewModels.

ViewModel design

For our application, the ViewModel is a central aspect of our MVC architecture. One of the first dilemmas facing MVC developers is to decide what the “M” in MVC means in ASP.NET MVC. In Rails, this is fairly clear, the M is ActiveRecord (by default). But in ASP.NET MVC, the “M” is silent! Its out-of-the-box architecture offers no guidelines nor advice on what the M should be. Should it be an entity? Data access object? DTO? Something else?

Sidenote – the term DTO is far overused. DTO is a Data Transfer Object. The pattern describes the usage, not the shape of a type. Just because an object is all properties and no methods does NOT mean it’s a DTO.

For us, the ViewModel is inextricably linked to Views, which leads us to our first rule:

Rule #1 – All Views are strongly-typed

I think I’ve gone over this one enough, as I really can’t stand magic strings and loose contracts. A dictionary as ViewModel is a very loose contract between the Controller and View. While on rare occasions we still need to pass information in the dictionary part, we’ve limited this to supporting information to help render some of the lower-level pieces of the View, and are used for some “plumbing” pieces, these pieces do not show up in our Controller action nor are they visible when you design the view.

Rule #2 – For each ViewModel type, there is defined exactly one strongly typed View

We’ll get into how we do this soon, but this rule has a lot of implications:

ViewModel types are distinct from our Domain Model types
The choice of what View to show can be decided strictly on the shape of your ViewModel
Re-used pieces in a View (through Partials) can be decided through re-using ViewModel types

On the first point, we never pass an Domain Model entity straight into the view. Most of the time, we only show a slice of information from a single entity. And many other times, the same snippet is shown in many places.

Rule #3 – The View dictates the design of the ViewModel. Only what is required to render a View is passed in with the ViewModel.

If a Customer object has fifty properties, but one component only shows their name, then we create a custom ViewModel type with only those two properties. Because we only have one ViewModel type per View, we shape our ViewModel around only what is displayed (or used) in that View. Why is this a Good Thing?

For one, just having a ViewModel points us to the right View. We need any other information other than your ViewModel type to pick the correct View. This also means we no longer need to concern ourselves with locating views by some arbitrary name, as the ViewModel is the View. Things like RenderPartial, which I have to select a name, become rather pointless at that point.

Rule #4 – The ViewModel contains only data and behavior related to the View

For the most part, our ViewModel contains only data. Most, if not all aggregations/calculations or shaping is done through our Domain Model. But occasionally, we have View-specific behavior or information, and that rightly belongs on our ViewModel.

We’ve looked at how we design our ViewModel and what it looks like, but how does it get there? If we create all these distinct ViewModel types separate from our Domain Model, didn’t we just create a bunch of work for ourselves? We thought so too, which is why we developed AutoMapper on this project.

Building the ViewModel

When we introduced AutoMapper into our MVC pipeline, we had a real problem. Do Controllers need to do the mapping between Domain Model and ViewModel in each action method? That becomes rather annoying for unit testing, as the mapping operation could warp things to a state that it becomes difficult to pull things back out. For example, or EditModels (ViewModels for forms) are very string-y, where DateTimes, Ints, Decimals etc are represented as strings. This comes from us using Castle Validators (future post, I promise) for validation.

So more moving parts, a dependency across all controllers? No, mapping in our Controller action just won’t do. Instead, we’ll use an Action Filter to do the work for us:

A request comes in, handled by an Action. The Action does its thing, ultimately deciding how to respond to the request. In many cases, this means rendering a view (ViewResult). From there, our Action Filter comes into play. On our Action method, we decorate it with an AutoMap attribute to configure the source/destination type pair to be mapped:

[AutoMap(typeof(Product), typeof(ShowProduct))]
public ActionResult Details(int id)
{
    var product = _productRepository.GetById(id);

    return View(product);
}

Very trivial, yes, but here we see that we still use the strongly-typed version of the View method, so that means that our model on the Action side, which I call the Presentation Model (feel free to pick a better name), is the strongly-typed ViewModel for the moment. The Presentation Model, which the Action creates, can be an entity, an aggregate root, or some other custom aggregate component that we build up.

From there, we decorated our action with a filter that specified we need to map from Product to ShowProduct. Why do we have to specify the source type? Well, many ORMs, including NHibernate, rely on proxy types for things like lazy loading. Instead of relying on the runtime type, we’ll explicitly specify our source type directly. This also helps us later in testing, as we can whip through all of our controller actions using reflection, and test to make sure the source/destination type specified is actually configured.

The filter attribute is very simple:

[AttributeUsage(AttributeTargets.Method, AllowMultiple = false)]
public class AutoMapAttribute : ActionFilterAttribute
{
    private readonly Type _sourceType;
    private readonly Type _destType;

    public AutoMapAttribute(Type sourceType, Type destType)
    {
        _sourceType = sourceType;
        _destType = destType;
    }

    public override void OnActionExecuted(ActionExecutedContext filterContext)
    {
        var filter = new AutoMapFilter(SourceType, DestType);

        filter.OnActionExecuted(filterContext);
    }

    public Type SourceType
    {
        get { return _sourceType; }
    }

    public Type DestType
    {
        get { return _destType; }
    }
}

We simply capture the types and delegate to the real action filter for the work. This is again because I believe in separating metadata in attributes from the behavior they perform. Attributes just don’t work well for behavior. Instead, I’ll create a separate action filter:

public class AutoMapFilter : BaseActionFilter
{
    private readonly Type _sourceType;
    private readonly Type _destType;

    public AutoMapFilter(Type sourceType, Type destType)
    {
        _sourceType = sourceType;
        _destType = destType;
    }

    public override void OnActionExecuted(ActionExecutedContext filterContext)
    {
        var model = filterContext.Controller.ViewData.Model;

        object viewModel = Mapper.Map(model, _sourceType, _destType);

        filterContext.Controller.ViewData.Model = viewModel;
    }
}

The BaseActionFilter is just a class that implements the various filter methods as virtual members, so I can override just the ones I need to use. The AutoMapFilter pulls the original PresentationModel out of ViewData, performs the mapping operation, and puts the mapped ViewModel into the ViewData.Model property. From there, the strongly-typed view for our specified ViewModel type is rendered.

Because AutoMapper can flatten source types, we often find our ViewModel to simply follow a property chain for various pieces of information. Again, we let our View shape that piece. If we decide not to flatten, it’s usually because we’re creating a partial that re-uses a ViewModel type across other parent ViewModel types.

Wrapping it up

Designing ViewModels is quite ambiguous with MVC, as the shipped platform doesn’t offer any guidance or opinions in that area. But by forming rules around our ViewModel, we can create a path and direction for our innovation. Our rules are designed to strengthen the relationship between the View and the Model, with a concept of ViewModel – a Model designed exclusively for exactly one View.

In the next post, we’ll look at designing our views – for both viewing and editing data, and how we’ve crafted opinionated HTML builders to eliminate a lot of duplication and enforce standardization.

Expressions Cheat Sheet

bogardj — Wed, 24 Jun 2009 19:36:53 PDT

I started getting really tired of looking up the translation between the ExpressionType and concrete Expression type (they don’t match up), so I created this cheat sheet that has each ExpressionType, derived Expression type and a simple example. What’s interesting is a few are only available in VB.NET, but no one really wants non-short-circuited Or/And operators, do they? You can download the PDF version, or just view the big list in all its glory:

ExpressionType	Type	Example
Add	BinaryExpression	int i = 2, j = 3; Expression<Func<int>> example = () => i + j;
AddChecked	BinaryExpression	int i = Int32.MaxValue, j = 1; Expression<Func<int>> example = () => checked(i + j);
And	BinaryExpression	Dim i As Boolean = True, j As Boolean = False Dim sample As BLOCKED EXPRESSION => c is int;

This is only for C# 3.0, the .NET 4.0 list should grow quite a bit.

The Filter-ViewData anti-pattern

bogardj — Thu, 18 Jun 2009 20:08:30 PDT

In just about every website you go to these days, its layout follows a very similar pattern:

You have some static resource logo, a large main section with information that our controller action supplies. But we also have some other stuff, those green sections. Things like breadcrumbs, shopping carts, login widgets, smart menus and so on. It’s all still data driven, but its information has absolutely nothing to do with what’s going on in my main section. This is some of the first kinds of duplication we see in MVC applications – common, data-driven sections across many pages in our application.

The usual way to get rid of this duplication is through ActionFilters. We decorate our main section action with filters:

public class ProductController : Controller
{
    [LoginInfo]
    public ActionResult Index()
    {
        var products = _productRepository.FindAll();

        return View(products);
    }

We don’t have that duplicated login information code in all of our controllers, and we can even go so far as to decorating our controller class or layer supertype to apply our filter to a broad set of controllers. But looking at our filter, it looks like we took a step backwards:

public class LoginInfoAttribute : ActionFilterAttribute
{
    public override void OnActionExecuted(ActionExecutedContext filterContext)
    {
        var currentUser = _userSession.GetCurrentUser();

        filterContext.Controller.ViewData.Add("currentUser", currentUser);
    }

Back again to magic strings! With MVC Contrib, it looks a little better as we can skip the key part, and just pass the current user object in. But beyond the magic strings issue here is the problem of linking what is essentially a view concern (how I organize the different pieces of information on a given page) with my controller structure. What if that widget isn’t shown on all actions for a controller? What if I have four widgets, and not all are shown on every screen? It begins to break down the normal, filters-on-layersupertypes-pattern. I’d rather not let my view structure, with master pages and what not, influence my main-line controller design in any way.

The other issue I run into is figuring out where the heck the information comes from if I’m looking at a view. I see a piece being pulled out of ViewData, but not any clear indication of where that information came from:

    <% Html.RenderPartial("loginView", ViewData["currentUser"]); %>

    <h2>View Products</h2>

Yes, we can put this in a master page. But now we’ve coupled the requirements of our view structure with the structure of our controllers, not always a good thing. In my (not-so) humble opinion, an action should be responsible for generating exactly one model. Things like action filters fiddling with view data are three layers of indirection to get around this rule. But in the MVC Futures assembly, we have a different option.

Tangential concerns with RenderAction

In our original view, if we dissected each piece, we had a main section and other, orthogonal sections. The main section’s information is handled by our main action handling the request. The other pieces are needed, but those concerns aren’t really related to the main section. Does showing a login widget have anything to do with showing a product list? No. But I don’t want to couple all the different concerns in completed view with the main action. Because those widgets, and where they’re located is a view concern, we can initiate the rendering of those pieces from the view using RenderAction.

RenderAction works like a mini-request, exercising the MvcHandler, but re-using the existing HttpContext. It simulates a request, but writes out to the exact same response stream, at the exact right place:

Our view doesn’t look much different, it looks like a cross between a Url.Action call and RenderPartial:

    <% Html.RenderAction<LoginController>(c => c.Widget()); %>

    <h2>View Products</h2>

What you don’t see is any data or model passed to RenderAction. The LoginController is a plain controller, with a regular action on it. The action now can be concerned with both finding the data for the model and choosing the view to render:

public class LoginController : Controller
{
    public ViewResult Widget()
    {
        var currentUser = _userSession.GetCurrentUser();

        return View(currentUser);
    }

I don’t have to jump through hoops to find where my information came from, as I only need to follow the link from the RenderAction call to the Widget action method. Additionally, my original ProductController now has absolutely zero logic for all of the orthogonal concerns, whether it’s hidden in an action filter or not. For each view, I can decide what master page to use, and place those RenderAction calls there. I let my view decide what layout is correct.

Wherever I see action filters poking around with ViewData, I see a very clear path to using RenderAction instead. It’s a much more cohesive and flexible model, and straightforward to understand where each individual piece comes from. Controllers have enough responsibility as it is, why do they need to be burdened with the additional responsibility of organizing data for tangential concerns? RenderAction has zero performance penalty, as it’s not a real, external request. ViewData can be abused as a bag-o’-stuff for views to consume, but that won’t lead to maintainable, understandable markup. Instead, we can utilize the RenderAction method and create very cohesive controllers and strongly-typed views, with explicit concerns.

More on Late-Bound Invocations with Expression Trees

bogardj — Wed, 17 Jun 2009 18:04:44 PDT

Recently, I went on a bit of a tear in the AutoMapper trying to improve performance. Besides the threading issues I introduced (dictionary lookups are NOT thread safe, in case you’re wondering), I looked at improving the performance of the reflection piece of AutoMapper. At its core, I map various kinds of MemberInfo objects on source types to MemberInfo objects on destination types. The semantic model is a little more involved of course, but that’s the basic model of going from one complex type to another.

I was quite dreading speeding up the reflection piece, not because IL generation and reflection emit isn’t interesting and all that, but it all looks quite ugly and hard to maintain. Until I stumbled upon Nate Kohari’s post on late-bound invocations via expression trees, I was avoiding that piece. The basic idea is that expression trees, when compiled, are able to use regular reflection MemberInfo objects to generate the right IL at runtime to do what you normally do at compile time. If you want a good sample of how to do IL, take a look at the expression tree compilation code in Reflector.

His example worked great for methods, but AutoMapper also supports properties and fields. To achieve the same result, I needed first to define a couple more delegate types to capture the kind of call you would do with fields and properties:

public delegate object LateBoundProperty(object target);
public delegate object LateBoundField(object target);

Next, I just needed to create the expression tree for calling a field and property. The property one is this:

public static LateBoundProperty Create(PropertyInfo property)
{
    ParameterExpression instanceParameter = Expression.Parameter(typeof(object), "target");

    MemberExpression member = Expression.Property(Expression.Convert(instanceParameter, property.DeclaringType), property);

    Expression<LateBoundProperty> lambda = Expression.Lambda<LateBoundProperty>(
        Expression.Convert(member, typeof(object)),
        instanceParameter
        );

    return lambda.Compile();
}

I first create an expression that represents a call to a property (the MemberExpression instance). Next, I need to wrap that MemberExpression call with a LambdaExpression, of type LateBoundProperty, so that I can execute the resulting delegate against any object I want. The one for a field is quite similar:

public static LateBoundField Create(FieldInfo field)
{
    ParameterExpression instanceParameter = Expression.Parameter(typeof(object), "target");

    MemberExpression member = Expression.Field(Expression.Convert(instanceParameter, field.DeclaringType), field);

    Expression<LateBoundField> lambda = Expression.Lambda<LateBoundField>(
        Expression.Convert(member, typeof(object)),
        instanceParameter
        );

    return lambda.Compile();
}

I still have a MemberExpression, but I use the Field method to create it instead. It’s a little func-y looking, but here is what that above expression would look like, if I knew the target type at compile time:

Expression<Func<Order, decimal>> expr = o => o.Total;

I have to jump through a few more hoops, as I don’t know the in and out times at compile time, it’s all stored as reflection information. The expression tree building doesn’t care anyway, as the above lambda syntax is merely syntactic sugar for what we wrote earlier. But how does this perform? As Rick Strahl points out, it only makes sense after many iterations. But I can then do fun things like this:

public class Order
{
    public decimal Total { get; set; }
}

[Test]
public void Sample()
{
    PropertyInfo propertyInfo = typeof (Order).GetProperty("Total");

    var lateBoundProperty = DelegateFactory.Create(propertyInfo);

    var order = new Order
        {
            Total = 50m
        };

    var total = lateBoundProperty(order);

    total.ShouldEqual(50m);
}

Since AutoMapper already has all of the reflection information cached on a per app-domain basis (that’s why you should only configure once), I can easily go through all of the reflected information and pre-compile all of the late-bound fields, properties and methods, so that they execute very fast during a mapping operation.

So what’s the downside? Startup is slower now, as it is CPU intensive to configure AutoMapper now. But compared to the startup time of NHibernate and StructureMap in our applications, it does not add that much more to the total startup time. Since we’re using AutoMapper in a web application, the we feel just as confident about its configuration time as we would NHibernate.

Unfortunately, expression trees in .NET 3.0 do not support setting fields and properties. But since that’s changing in .NET 4.0, I think I’ll just wait to optimize the destination member assignment part of AutoMapper until then.

Why opinionated input builders for ASP.NET MVC?

bogardj — Mon, 15 Jun 2009 07:00:00 PDT

When we first started looking at crafting forms for MVC in a recent big project, we had just completed a ton of “view” screens for a wide variety of information. Having learned quite a few things on crafting HTML on the view side, we had a few goals in mind to ensure that we improved our development experience on the edit side.

But it wasn’t just the development experience we wanted to improve, it was the design/conception/analysis phase as well. By centering our conversations around creating “inputs for” our view model, we could create a lot of consistency in our screens. Our major goals for our opinionated builders were:

Input elements should look the same
Common classes of input elements should look the same, so that enumerations are always in a drop-down, nullable enumerations are in a radio button group, date-times are in a special date and time template, etc.
Our UI designers should not have to think about designing individual input elements
Developers should not have to think about what the standard input element HTML is
Everything rendered should work with ModelState, validation, and UI testing in a completely consistent, predictable and deterministic manner

In many of the MVC examples out there, I see a metric ton of duplication around designing forms. You always see the same “label” element, the input element, validation errors, asterisk for required fields, and so on copied over and over again for every single form element. What a waste!

Removing the duplication

What we saw was that if we gathered information from the strongly-typed expression, we could make an informed decision on what HTML to render. Is your model member decorated with a required field attribute? Render an asterisk. Is the type of the member an enumeration? Render a drop-down, but include all of the other normal input element business (label, etc.). Is the type of the member a Guid? Render a hidden field. But funnel all of these decisions through a single call to a single method, “InputFor”.

The benefit of opinionated input builders was that it was just one less design decision that had to be discussed and implemented, all the way through the process. By employing rigorous standardization on our design, we could create a very consistent user experience.

I don’t really see a panacea for a master opinion framework, as you’ll still have to decide what opinions are valid on your project. But ASP.NET MVC, like many other frameworks, is one that allows you to form your own opinions, without making any recommendations on its own. Do make sure you form these opinions, as otherwise you’ll be stuck maintaining a lot of duplication of HTML and behavior in your views.

Requesting features for AutoMapper

bogardj — Sun, 14 Jun 2009 18:08:09 PDT

On the AutoMapper mailing list, I get a lot of what I consider wacky requests. Not because the requests aren’t valid, but rarely do I get any context of what people are trying to do with AutoMapper. As a reference, we are using AutoMapper in these situations:

Mapping from Domain/Presentation Model to ViewModel
Mapping from Domain/Presentation Model to EditModel (forms) <- yes, different concerns at play here
Mapping from Domain to a Reporting model (think printable view screens, not analytics)
Mapping from EditModel forms to Command objects

Some of the requests I’ve inferred include a lot of mapping from ViewModels back to Domain models. Personally, this seems wacky to me, unless you’re in some sort of ActiveRecord/CRUD scenario. In those cases, I’d rather expose my model directly to my views.

If AutoMapper doesn’t do what you want, a few things might be true:

There is a bug
You’re doing something valid, but in a scenario I haven’t encountered
You’re doing something you shouldn’t do, and trying to attach two boards with a screw and a hammer

Object-to-object mapping is a broad spectrum of distinct scenarios, and AutoMapper really only focuses on the scenarios where your destination type looks like your source type, plus some flattening. So if you’d like a scenario to be supported by AutoMapper, two things would greatly help me:

Code showing a failing test
Description of the context for which you’re trying to use it

Context is very helpful for me in the overall design, as only failing tests show me contrived, Foo to FooDto scenarios. It helps if I understand at what place in your architecture you’re trying to use it, as it may not line up to how it was originally designed. But context tells me if there is a broader concept at play, and I can design (or not) for that category of usage.

Thanks again to everyone that has contributed, I’ve been consistently surprised by the number of feature requests that include a patch. To date, I’ve received around 25 patches from the community, far exceeding any expectations I had.

Simplest versus first thing that could possibly work

bogardj — Wed, 10 Jun 2009 19:58:21 PDT

One of the core XP practices that resonated with me quite early on was the concept of simple design. When I learned TDD, this practice was further refined with the concept of doing the “simplest thing that could possibly work”. To make a test pass, I would code the simplest thing that could possibly work. It takes quite a bit of discipline to adhere to this mantra of simplicity, fighting a constant urge to design something more complex than the problem at hand requires.

Browsing the XP wiki, you can find a lot of discussion of what exactly this means. TDD calls for “Red, Green, Refactor”, which might lead you to wonder why you would need to refactor after doing the simplest thing that could possibly work. It seems that the consensus formed around first performing the fewest steps, then refactoring to the fewest pieces or components. But in our quest for simplicity, I notice a second, more subtle mistake: confusing the first thing that happens to work with the simplest thing that could possibly work. If I choose the first thing that happens to work, my refactoring step often leads me merely to the simplest solution, but not the most elegant.

The difference between the two is easy to fix – it just requires thinking! Thinking about possible solutions, different designs, vetting alternate paths to the goal. This can be accomplished through pair programming, whiteboarding, and just about any exercise that requires us to think of at least two possible solutions before picking the winner. I don’t necessarily see this happening with every possible solution, however. But some of the most awkward designs I’ve created seem to stem from just picking the first thing that works, and not doing a little thinking.

Which is quite sad, as a little effort and investment in investigating multiple designs pays off many-fold in the long run. Evolutionary design works best when we’re not stumbling in the dark, but making informed decisions with the most options as possible on the table. If this sounds something like the idea of concurrent or set-based engineering, it should! Except in this case, we perform it in the micro, at the point of every non-trivial design decision. Simplicity is not automatic, and often comes from choosing the best design from a few options. But we can’t be lazy about it, as taking just five minutes to think of a different approach can save man-hours (or days even) further down the road.

Defining and refining conventions

bogardj — Tue, 09 Jun 2009 06:35:34 PDT

At last night’s ADNUG talk, Jeremy Miller talked about Convention over Configuration, and many of the principles the Rails community embraces. He showed a few examples of opinionated software, such as FubuMVC. One thing I would have liked more conversation around (but no time, alas), was the process of discovering conventions and forming the opinions that make up opinionated software.

Opinionated software, as I see it, is a framework that provides intentionally chosen axes of change, where other axes are fixed to adhere to agreed-upon principles. In Rails Active Record for example, by default my entity’s shape is whatever shape my table structure is, optimizing for very little configuration. Because Ruby is a dynamic language, it can get away with this easily with meta-programming tricks.

But how do we arrive at such opinions? How do we decide which principles are acceptable, which are not? Every design decision has a tradeoff, and frameworks like Rails aren’t going to satisfy everyone’s opinion. We need some mechanism to form these opinions and craft our conventions.

Integrating evolutionary design

With our wall of pain, we strive to ensure that we have one design vision. Introduce a refactoring, and we want to retrofit and remove duplication everywhere. Often, this is architectural duplication, such as the knowledge of a required field propagated throughout every layer in the system. To eliminate this architectural duplication through conventions and opinions, it would likely take several iterations of that design before everyone is happy with the result.

But the cost of propagating that design does have a real, tangible cost. Iterating a design along with propagating it will cause a very real churn, even to the point where it could frustrate developers and discourage innovation. On the flip side, iterating endlessly and never retrofitting our opinions leads to chaos, as well fall into the trap of having a hundred truths in our system, all of them correct at one point in time. A new developer that came on recently vented his frustration with this problem, as he was spinning his wheels on an old design, simply because he picked the wrong version of the truth to model from.

So we need to both iterate and propagate our design, ideally at key tipping points where we’ve arrived at a sound design, and no important unanswered questions remain. We might have questions about our design, but answers might only come through applying our design in a variety of scenarios.

Past the tipping point

From my experience, these tipping points are fairly obvious, and follow Evans’ concept of breakthrough refactorings. We make incremental enhancements, slowly improving our design over time. At a critical mass of awareness of problems and understanding of the domain, we introduce a change that dramatically improves the design.

These tipping points can’t be anticipated, nor can they be prognosticated. In fact, trying to form opinions in absence of any context for these opinions is very likely to lead to awkward, friction-inducing development. Another word for premature opinions might be BDUF.

The middle ground here is one where we become finely attuned to the pain induced by our design, not try to invent problems where they don’t exist, iterate our design, and retrofit after each breakthrough. Opinionated software is a fantastic concept, but we can’t confuse opinion formation with misguided attempts to make all design decisions upfront in the absence of agreeing upon the principles that led to the opinions.

Reflecting reality

bogardj — Sun, 07 Jun 2009 20:18:09 PDT

Reading over the latest MSDN magazine issue, I’m always encouraged when I see something that I consider important on the cover, Test-Driven Design. It covers one of the more difficult technical aspects of TDD, which is mock objects. It took me a couple of years before I really understood the “right” way to use mocks as a design tool, and it wasn’t until I read the Meszaros book that the concept of direct and indirect inputs and outputs laid down simple rules for mock objects.

The really strange thing about this article is that it uses NMock as its mock object library. NMock? Nothing against the library, but I can count at least two mocking frameworks that are more popular, Rhino Mocks and Moq. From folks I talk to in the ALT.NET circles, it’s the vast majority of folks using Rhino Mocks, with a healthy following of Moq. Folks using NMock are those stuck on older codebases, as NMock is (as far as I can tell) the earliest mocking framework for .NET. It’s all well and good to show one mocking framework, if that’s the one you’re accustomed to, but to not even mention the most popular mocking framework in .NET? A little strange.

So what’s the real issue? Here’s some example code from the article:

[Test]
public void ReceiptTotalForASaleWithNoItemsShouldBeZero()
{
    var receiptReceiver = mockery.NewMock<IReceiptReceiver>();
    var register = new Register();
    register.NewSaleInitiated();

    Expect.Once.On(receiptReceiver).Method("ReceiveTotalDue").With(0.00);

    register.SaleCompleted();
}

This is frustrating, string-based mocks? Here’s a fun TDD exercise, rename the ReceiveTotalDue method, and watch how many tests break for the wrong reason. Tests should fail because of assertion failures, not because I renamed a method. Additionally, the Arrange-Act-Assert syntax is nowhere to be found. The AAA syntax made mocking fun, and for me, useful, as the record-replay model led to brittle, difficult to read tests.

Digging deeper, it turns out that the article was written by a ThoughtWorker, and ThoughtWorks sponsors the development of NMock. I’m glad that mocks got some love in MSDN, but I’d rather the article reflect today’s reality of using mocking frameworks.

Fighting technical debt with the wall of pain

bogardj — Wed, 03 Jun 2009 17:57:39 PDT

Technical debt, even on the agile-ist of agilista teams, still accumulates. Debt is inevitable, as initial design is always based on assumptions, not all of which pan out. I never view that as a failure, as all we can do is make the best decision given the limited information we know at any given time. Experience with DDD has shown me that often, design and architecture doesn’t undergo linear improvements. Rather, we reach a critical mass of information and knowledge where it becomes plainly and painfully obvious that our current design will no longer suffice.

In other situations, technical debt arises out of creating hidden forms duplication where only large-level, architectural refactorings can take the codebase to another level. This duplication comes not necessarily from domain concepts, but structural concepts, or what I like to call the “application domain”. We often need to create a model of our application, with things like the Command pattern, Strategy pattern and so on. Breakthroughs in this area happen just as they do in our model of the problem domain, but instead arise out of use cases and the structure of our application, rather than from a deeper understanding of our problem domain.

Finally, we have technical debt arising out of finding better ways of accomplishing the same result. It could be a layer supertype, switching to a jQuery plugin, or just plain removing subtle copy-paste duplication. Often, we see this last kind of technical debt arising out of larger teams, where individual frequency of duplication is rare, but project-wide duplication is far more pervasive. For example, I might introduce subtle duplication around an area once every three weeks. Because it is so sporadic, I don’t necessarily perceive the problem. But if I have five other team members, the frequency of duplication might be three times per week, above the threshold for one developer’s perception.

To tackle the inevitable growth of technical debt, we need some sort of strategy in place to address that risk.

Option #1 – Do nothing

Yes, the team might say a lot, complaining and the like, but never do anything about it. Or, they are simply not aware. Ignorance, apathy or hopelessness, I’m quite sure this approach is not viable for applications that need to change.

Or, the team has made management or the product team aware of a problem, but were unable to explain or justify the work required to fix the problem, so it merely lingers. I’ve been in this option far too often in my career, and it’s never a fun feeling.

Option #2 – Cowboy style

In the absence of any form of technical story artifact, it’s up to the development team to prioritize technical debt payments, in the form of refactoring stories. Outside of any strategy, prioritization and planning is often cowboy-style, done immediately when found, without regard to impact or other more important issues. Refactorings are chosen based on how interesting it might be, the level of immediate frustration of the developer finding the duplication, or how easy the fix might seem.

Most of the time, it’s done exactly when the fix is found, without much effort into analysis of the impact, cost or benefit of the fix. Additionally, the fix is usually quite local, and not propagated to the rest of the system. We might make it easier going forward working with a standard component, but all too often this new standard is not retrofitted back to the rest of the application. This leads to multiple “right” ways of doing things, where the design of the next new component starts from the last component built, as long as we happen to remember what the last component was.

This happened on a recent project, where we created a new layer supertype, and all new controllers needed to implement this new base class. Unfortunately, this design was refined several times, until we had a half-dozen versions of the “right” way to design a controller.

It’s quite enticing to cowboy-code technical debt fixes. We, as the developer, get to play the hero type, announcing to the team “I fixed the Floogle problem!”. It’s a good feeling, unless you’re met with blank stares and unspoken (or vocal) questions of “why did you waste time on that?”

Option #3 – Wall of pain

Instead of fixing technical debt issues at the exact moment we encounter them, our team instead keeps a prioritized queue of technical debt items on our whiteboard. On it, we track two sets of items:

Problems with a proposed solution
Problems with no solution, and a plea for help

As a contract to ourselves, we took an oath to not go forward with a new solution without retrofitting the application to use the new concept. This ensured that we didn’t have pockets of bad code intermixed with good code, or worse, seven different designs in our system of the same concept. The retrofitting was key for us to continue to innovate and improve, and let us find broader concepts. The more examples we had of a concept, the more we could find commonality. If there are seven ways to do the same thing in our system, there is just as much duplication, but a duplication that is much much harder to both recognize and fix.

Kevin and Jeffrey teamed up to imbue in me a sense of pain-driven development. The more something hurts, the more it needs to be fixed. If local builds take 10 minutes, that’s a lot of pain throughout the day. This concept led us to our current “wall of pain” that includes these items:

The problem
The solution
The level of pain (1-5)
The estimate to retrofit, in points

If something is very painful, but easy to fix, it is moved up on the list. However, the effort to retrofit has to be a consideration, as it often indicates the risk introduced into our system of a large change. Since a defined solution is part of this list, the point estimate is defined in terms of relative complexity, and therefore risk.

For items without a solution, we just leave out the solution and estimate to retrofit. These items are analyzed and given a solution as needed. Many designs need vetting before established as the “new standard”, so our team will wait to pick a solution until we’re quite satisfied with the result.

All of this assumes that technical debt items aren’t tracked in your normal taskboard/kanban/story wall. Otherwise, you’ll likely have an established manner of prioritization.

We’ve had our wall of pain up for a few months, and at the very least, it provided visibility and a way for us to communicate the day-to-day development pain we might face. We still introduce minor fixes and technical debt payments during our iterations, without using the wall of pain, but these are of the small and completely obvious variety. Still key is the concept of retrofitting, which we find greatly helpful in maintaining a consistent codebase.

Meaning of life

bogardj — Thu, 28 May 2009 18:48:48 PDT

Wolfram Alpha knows all:

From OMG to OMFSM, we now move to OMWA.

PTOM: Black-box analysis of legacy applications

bogardj — Wed, 27 May 2009 20:20:33 PDT

It seems like such a great situation, you’ve been tasked with replacing an old legacy system with a shiny new Web 3.0 AJAX-ified replacement. It’s your chance for the limelight as the knight in shining armor to come charging in with a new system that will *surely* blow away the old one. That is, until you get mired in the problem of requirements coming in of “make it work like the other system”.

As a consultant, it can take quite a while to extract what “success” means, especially as you might have so many different roles with widely differing opinions of what the new system should do. The CIO wants whatever the buzzword of the month is, probably SOA or “Services”. The sales team wants something “frickin sweet” and “snazzy”, while the IT department likely hates your guts, as you’re replacing the system they built. Talk to the people actually using the existing system, and they probably hate it, but don’t want to change. Listen solely to them and you’ll get another mainframe application, developed in .NET.

With a large existing system, replacing what already exists can’t come in one fell swoop. Often, you’ll need to carve out subsystems of functionality, each of which likely needed its own system to begin with. I never fault the original developers of these systems, as many were built at least 10 years ago, some 20. It’s hard to build an application to last 20 years, especially if your business grows beyond what anyone might have imagined.

Back to replacing a legacy system, we have to understand what the existing Godillapp does. Recently, I worked on a project to replace a large chunk of a huge legacy system. Some stats of the existing codebase were:

1 million lines of SQL
1.2 million lines of COBOL….NET
800K lines of VB.NET

This system did everything and was the lifeblood of a billion-dollar corporation. Of course, it wasn’t a billion-dollar corporation when the system was first written 20 years ago, and “replaced” 5 years ago, but it did take the company a long way. As a part of this experience, we gained a few nuggets and strategies for dealing with these types of engagements in the future, and what follows is a list of those lessons learned.

Lesson #1 – What it does versus how it’s used

When we talked to various people about this application, we encountered two different stories. One was from stakeholders who created the application, and one was from the people who used it. The people who created it could tell us exactly what the application did. They talked about major areas, features, database tables, screens and so on.

But one thing was missing – how did people use the system? For that, we talked to the actual end users.

Which showed a completely different story. There was a lot of “swivel chair” integration going on, and users’ cubicles were covered in little post-it notes, cheat sheets and reference guides to serve as a living instruction manual. Does that customer have a discount? Well, put a note in the order comments field so that Cheryl in accounting remembers to note that in the invoice. These insights into end-user behavior gave us knowledge that merely looking at the application, code and database would have obscured and even misled. This, combined with input from all stakeholders, along with vision of where the company wanted the application to go, gave us direction into the system we needed to build.

But we still needed to replicate existing functionality.

Lesson #2 – Craft your own story of the data

The application’s database was about the most interesting I’d ever seen. It was ported from a mainframe, pretty much straight over. This meant:

Lots of fixed-width columns
No referential integrity
Evolving architectural style

For the core tables, we might have OLD_NMNG_OF_CLMNS, where as new tables followed tbl_hungarian notation. To get around this, we exported the schema into an entirely new database, and created associations between tables as we found them. We didn’t bother with all of the original columns, just associations, so that we could understand what related to what. We couldn’t modify the existing database, but a playground of just the schema let us create reference diagrams of the “real” relationships of the tables.

Next, we created searchable schema. We created a table that included:

Database name
Table name
Column name
Column data type

Yes, I know this was available from the schema views, but we needed something that was super simple to query from. With a system of hundreds and hundreds of tables, it helped to do things like “SELECT * FROM dbSchema WHERE colname LIKE ‘%PRODUCT_NUM%’”. We didn’t know what was related to a product, but we could make an educated guess. Since the original database had zero foreign key constraints and thousands of stored procedures, we needed to be able to take large guesses of how data was related to each other. Existing developers might be able to give us direction, but with a system this big, we could only really rely on the actual system.

Lesson #3 – The data don’t lie

This one’s from the phrase “ball don’t lie”. You come to the business owners and ask, “what shipping codes do we need to support?”. The answer is “all of them”. There aren’t that many shipping services in the world, yet you find a hundred different ship codes in the database lookup table. Do you really need to support all of these shipping codes? Probably not.

But to answer this, we need to gather more information on what these values mean. Are some of these just test shipping codes that some developer put in the system six years ago? Are some for obsolete shipping methods? Again, we turned to analyzing the existing data. It took a long time to run this batch process (around 4 hours), but we crafted another table that contained:

Column name
Column value
Number of occurrences
Percentage of total
Last date of value

Essentially, we would group a table by a column, one by one, gathering statistics on how much each value was used, and when it was last used. This gave us the hard data that said, “see that code ‘OC4’? It was last used 5 years ago, and only twice”. With that information, we could dive in to the original database, and find that the person who created these orders was “Cosmo Kramer”. With this information, we could figure out exactly what kinds of things we needed to support.

Additionally, we were able to provide hard data on who used what features. We might find a certain shipping code used only 100 times, a fraction of the total, but these were used by one of the company’s biggest customers, so it suddenly jumped in importance. It was still the customer who decided what to support, but we were able to give them a much, much better picture of the data than many of their seasoned veterans did not know.

If you can’t run a big batch data mining process against the database, I suggest you learn two SQL techniques: the GROUP BY clause, and derived tables. The general idea is to first create an inner query where you group by values, and then an outer query that performs additional grouping and statistics. This gives you quick information like “what percentage of total orders used the OC4 shipping code?” Very useful information for decision makers.

Lesson #4 – Profile, profile, profile

In a codebase that’s large enough, debugging code is counterproductive. Instead, we can take the route of poking an application on one side, and seeing what comes out the other, while listening in to the conversation using a SQL profiler (or network profiler). In this manner, we can do things like add a line item to an order, and see how the application performs that operation. In one example, we added a single line item to an order and over 280 queries executed to perform that one operation.

We might find common stored procedure names, and go back the code armed with a textual search to see exactly what code was executed. While the SQL codebase had over a million lines, profiling told us that only a fraction of this was actually used. Because there was no database change management system in place, we saw a hundred different sprocs to retrieve an order. Instead of using an existing sproc, it was easier just to create a new one.

As we profiled more and more, we combined this knowledge back into our metadata database, adding comments to what the purpose of each table was. The list of “important” tables was eventually whittled down to what you might expect, and many were just a bunch of features that never panned out, cancelled projects and so on. With all of our information, we could almost look backwards in time to see the entire history of the company.

Lesson #5 – Dump the negativity

Although it’s easy to trash-talk the existing system, it won’t make you any friends with the existing IT folks. No one likes having their baby called ugly, even it fell off the ugly tree and hit every branch on the way down. You don’t know the political situations for the last 20 years, nor the different forces at play. These systems put a lot of food on a lot of people’s plate, but no longer help the company grow. The only thing you can criticize is a failure to recognize when the existing system no longer matches the company’s needs. Other than that, best to keep positive, stay focused on delivering value, and just *quietly* make notes for future Daily WTF contributions.

Wrapping it up

Replacing a legacy system is costly and risky. Anyone involved with projects like these can attest to the frequency of failures in this arena. But armed with a well-thought out strategy, we can maximize the possibility of success. Nothing’s guaranteed of course, but at the very least, we’ll have vastly increased the knowledge of the existing system.

Subverting TDD as a design tool

bogardj — Fri, 22 May 2009 06:29:45 PDT

So TypeMock can now mock DateTime.Now. Replacing the functionality of DateTime.Now, which is an external dependency, was one of the first issues that taught me the value of TDD. With TypeMock replacing DateTime.Now, I get all the benefits of unit testing, but none of the benefits of TDD. Yes, my code is now “testable” in the sense that I can now write a unit test against my code, but I’m using Jedi mind-tricks to do so.

TDD combines example-driven, client-first development with the icing of providing a safety net of providing executable specifications to lock down existing behavior. For me, the real benefit of TDD is the former, much more than the latter. TDD tells me exactly where my design is bad, as tests that are hard to write belie a design that is hard to work with.

When I first hit the “DateTime.Now” problem, TDD led me down a path that forced me to invert dependencies. Instead of an opaque dependency on DateTime.Now, I had an explicit relationship between that class and its dependency through an IClock interface:

public PunchClock(ISystemClock clock)

From the standard external system dependencies such as clock, files, configuration and so on, I’ve used the dependency inversion principle everywhere inside my codebase. Why? TDD and DIP tell me if my class is doing too much. If my class is doing too much, it will be hard to understand, change and maintain. In my experience, TDD is by far the most efficient tool at showing me deficiencies in my design. It shows me not only problems in design of individual members and even type and member names, but problems of my overall architecture.

If I have a legacy system I need to change, there’s a whole book on techniques for doing so in a safe, responsible manner. Changing legacy code is like camping – you always leave your campsite cleaner than when you found it. But I still can’t understand why I would need a tool that subverts all of the indicators in a unit test of bad design.