Alex Scordellis

RESTful architecture: what should we PUT?

2010-11-04T11:18:00.001Z

I've recently been reading REST In Practice, by Ian Robinson, Jim Webber and Savas Parastatidis. I had some knowledge of the hypermedia-driven architectural style, but this book has really helped me clarify my understanding of both the how and why. It's also convinced me that I should definitely consider Atom for event-based integration requirements.

There was one concept I found puzzling. In Chapter 5 (the callout box on page 114, if you have the book), the authors recommend using POST to update the state of a resource. This is driven by a choice of interpretation of the semantics of PUT. According to the HTTP spec:

If the Request-URI refers to an already existing resource, the enclosed entity SHOULD be considered as a modified version of the one residing on the origin server.

In the book, the authors interpret this to mean that the body enclosed with the PUT request should contain the same elements as the representation served by GET requests at the same URI. Since we are using the HATEOAS style, representations include links and other hypermedia controls. The implication is therefore that clients should PUT representations containing both business data (for example the new contents of the coffee order) and hypermedia controls (the available next steps in the workflow). To quote the book:

this obliges a client to PUT all the resource state, including any links, as part of the representation it sends

The problem here is that the client has no business determining what workflow steps are available. It's the server's job to understand what steps are available given the current resource state, and advertise those to clients using hypermedia controls. Therefore, we don't want the client to send the complete representation including both data and hypermedia controls.

I see four potential resolutions to this conflict:

Use PATCH: The PATCH HTTP verb is designed to explicitly support partial updates to a resource. However it's not widely supported. It also feels semantically wrong to me. From the client's point of view, the business data comprising an order (how many lattes?) is the whole resource. The client doesn't see this as a PATCH, but as a replacement, i.e. a PUT. PATCH might make sense to express concepts like "use skimmed milk in the latte, instead of the full fat I originally ordered".
Use POST: This is the approach suggested in the book. For me, it has similar drawbacks to PATCH. POST implies appending to a resource. POSTing one cappuccino to a coffee order resource feels like it should add one cappuccino, not replace the existing set of ordered coffees with one cappuccino.
Use PUT, including hypermedia: To follow the strict interpretation that PUT should include entire representations, the client sends both the entire new coffee order and whatever hypermedia controls the service last sent it. The service then ignores these controls, since it's the service's job to determine what they should be, and on future GETs the service sends whatever controls it deems appropriate at that time. This feels nasty; we are sending unnecessary data just to satisfy some architectural OCD (hat tip to Seb Lambla for that phrase!).
Use PUT, don't include hypermedia: The client sends a complete representation of the new order, but no links. To me, this feels conceptually right. The client fulfils the expectations of PUT by sending a complete representation of the parts of the data for which it is responsible, but does not pretend to be responsible for determining what hypermedia controls are available.

I explored some of these thoughts through a twitter conversation with @serialseb, @iansrobinson and @jimwebber. Being able to explore these thoughts in conversation with some of the experts is incredibly rewarding, compared to sitting alone pondering, so thanks to those guys for their contributions. Ultimately we came up with a simple rule of thumb, which seemed to attract mutual agreement:

In response to GET requests, services serve complete representation of the current known state, including business data and available hypermedia controls. Clients PUT complete representations of the parts for which they are responsible.

HTTP places two significant expectations on PUT requests: idempotency, and the concept that the enclosed entity-body is a complete representation. This rule of thumb satisfies both, while absolving clients of the need to include data over which they have no control or responsibility, provided we accept that GET and PUT representations need not be the same.

This raises another question. Available hypermedia controls are just one example of representation elements that only the service should be generating. Other examples include service computed values (for example the total cost of a coffee order) and resource state that is owned by the service (for example, whether or not the order has been paid for). We don't want the client updating the total cost, or sending state fields telling the service that the client has paid, when it hasn't. Either of those could be bad for business! Should these data items be included in client PUT requests? My feeling is no. According to the rule of thumb, clients only include the parts for which they are responsible. However I have a feeling that this may be a controversial standpoint. From the twitter conversation, I learned that the authors aversion to partial PUT came from none other than Mark Nottingham.

Disclaimer: I hope that I have not misrepresented the viewpoint of the book and its authors. If I have, I apologise and welcome feedback and corrections.

Firefox autocomplete plays havoc with hidden input fields

2010-06-01T18:17:00.001+01:00

We recently had a bug in our system which we eventually traced down to some odd Firefox behaviour. We’re using hidden input fields in our HTML to pass various pieces of data from the server to JavaScript code. We found that on some occasions, these hidden input fields would have different values from those sent by the server, even before any JavaScript that could change the values had run.

We realised that Firefox was using its autocomplete feature to auto-populate the hidden input fields with old values. The problem would only happen when some JavaScript had set the value of the hidden field, then the user pressed F5. The value set by the JavaScript would then reappear after the reload, even though the server sent a different value.

The solution we found (credit to Jim) is to use the non-standard HTML attribute autocomplete="off". You can also use an invisible span. The following page demonstrates the problem:

<html>
    <body>
        <input type="hidden" value="original" id="broken"></input>
        <input type="hidden" value="original" autocomplete="off" id="fixed"></input>
        <span style="display: none;" id="span_not_affected" value="original"></span>
        <script type="text/javascript" src="http://ajax.googleapis.com/ajax/libs/jquery/1.4.2/jquery.js"> </script>
        <script type="text/javascript">
            alert("input with no autocomplete attribute: " + $("#broken").attr("value"));
            alert("input with autocomplete attribute: " + $("#fixed").attr("value"));
            alert("span: " + $("#span_not_affected").attr("value"));
            $("#broken").attr("value","changed");
            $("#fixed").attr("value","changed");
            $("#span_not_affected").attr("value","changed");
        </script>
    </body>
</html>

Save this page as an html file and open it in Firefox. You'll get 3 alerts saying 'original', as you might expect. Reload it (F5), however, and the first alert, will say 'changed'. The other two will not be affected. Firefox has remembered the value 'changed' that was set by the JavaScript the first time round and ignored the value in the source HTML.

I’ve seen this behaviour in Firefox 3.6, but not in any of Chrome, Safari or IE6.

So, er, which build label do I deploy this week?

2010-03-24T17:56:00.001Z

The problem

A couple of weeks ago our team accidentally deployed the wrong build to our production environment. We deployed that latest green build from CruiseControl.NET, rather than the slightly older one that we’d intended. Fortunately, this didn’t matter too much. After all, this build had passed our test suite, so there were no serious bugs.

We decided to perform some root-cause analysis, using the 5-Whys technique, to understand how this had happened. We came up with a few root causes:

It’s hard to tell what code changes are contained in a given build. The only way to do this is to analyse the git commit history, which can be rather tangled. To make this easier, we have now agreed on a convention for git branch naming and merging patterns, based on this one from @nvie.
We were testing stories, not builds. This is a natural way to approach a story-driven Agile project, but stories aren’t what gets deployed to production, builds are. Come time to deploy, we’d know that stories #123, #134 and #96 are signed off, but that sign off would be tied to a point in time, not a build number. How do we know what build to deploy for these stories?
We didn’t have a way to retrospectively ‘fail’ green cruise builds. Just because CruiseControl.NET passes a build, it doesn’t mean it’s perfect. When a QA finds a bug during manual or exploratory testing of a story, they fail that story. But we really need to fail that build. We don’t want to deploy this build or any following one until this bug is fixed.

The solution

We came up with a great, low-tech way to solve this problem: coloured stickers!

When the developers think that they’ve finished coding a story and that it’s ready for test, they put a white sticker in the bottom left of the story card, and write the build number on.
If QA pass the story, they stick a green sticker on, with the build number against which they tested written on it. The story is now ready for deployment.
If they find a problem, they use a red sticker. Again, they write on the sticker the number of the build in which they found the problem.
When the developer commits their fix to the problem, they use another white sticker labelled with the build number containing the fix. This will be followed by another green or red, and so on.

What does this give us?

Imagine I want to deploy stories #123, #134 and #96 to production. We know that they’ve all been passed by QA, but we need to answer some more questions.

What build contains working versions of all these stories? It’s now very easy to answer this: just look at the green stickers on each card. To deploy these stories, we need the latest of the build numbers on the green stickers.
Is the build I’m deploying safe to deploy? Anything on the wall with a white or red sticker not followed by a green one represents a build containing code that either is known to have a bug or has never been tested. Let’s say we want to deploy build 2503 (it has a green sticker for a story we want to deploy to our users), and the currently deployed build is 2496. If we see a white or red sticker on a story that’s still in test with build 2499 on it, we know that we can’t deploy 2503. The team can now focus on finishing the story that’s carrying the sticker for build 2499, so that we can get these features to our users as quickly as possible.

What’s great about this is how easy it is to glance at the wall to get the answers to these questions. We don’t have to go digging in our cruise result logs or story tracking tool or the project manager’s spreadsheet.

Other benefits

Each time a story goes into test and fails, that is waste. In the language of lean thinking, we see rework (QA may have to retest areas of functionality after the fix, dev may have to re-code something), transportation (extra handoffs between QA and dev, often delayed if one party is busy working on something else) and waiting (QA and dev waiting for each other to be available for collaboration, dev idling while a story is QAed as they don’t want to pick up a new one and thus have to switch context back to the one in test). We are always looking to minimise waste, so we’d like to reduce the number of trips taken by each story around the dev-QA cycle.

The number of stickers on each card is an immediate visual indicator of this metric. A quick look at the wall (again, no digging in the story tracking tool!) shows which stories have incurred this type of waste. We might then decide to apply some root-cause analysis and other continuous improvement activities to those stories, so that we can understand what really caused the waste and prevent it in the future.

I can’t recall whose idea this was, so credit goes to the whole team. Extra credit to Nick Ashley, who chose the type of stickers to use when I was telling him to use a completely different sort that would have been entirely impractical.

Lambda-passing style for access to resources with lifecycles

2010-03-16T18:28:00.001Z

When accessing a resource whose lifecycle must be carefully controlled, such as database or network connections, we would traditionally use a try..finally or using block in C#. Something like this:

using(var db = connectionPool.GetConnection())
{
    results = db.ExecuteQuery(query);
}

DatabaseConnection db = null;
try
{
    db = connectionPool.GetConnection();
    results = db.ExecuteQuery(query);
}
finally
{
    db.Close();
}

I'm not really a fan of either of these styles. The try..finally one is particularly nasty. It would be very easy to forget to call db.Close(), or to forget to put that call in a finally block. It’s also unpleasant to have to declare and initialize the db variable outside the scope of the try block. Not only is it ugly, but it means that code further down could try to access the database connection once I’ve closed it.

The using option isn't quite so bad, but as a client of the connection pool (or whoever is managing my resource), I still have to remember to call Dispose() or wrap it in a using block. It also requires that the DatabaseConnection type expose both the operations (e.g. ExecuteQuery) and the disposal mechanism, which may more naturally belong on the connection pool.

I prefer the following style:

var results = connectionPool.WithOpenConnection(c => c.ExecuteQuery(query));

With this style, there's no way we can forget to close the connection. There's also no way that we can access the connection object outside of the intended scope.

The code inside my connection pool object usually looks like one of the examples above, using using or try...finally. I've now managed to confine the use of those unpleasant constructs to just one place in the codebase, rather than repeat them every time I need to use my database.

Typically, WithOpenConnection has a generic type parameter to specify the return type. Because C# doesn't allow void to satisfy a generic type parameter, you typically need to provide two implementations of WithOpenConnection, one for when your operation has a return value and one when it doesn't. This is the only drawback I can see of this approach.

Building Quality In

A common theme in lean and agile software delivery is to build quality in. That is, rather than inspect our software for bugs after we’ve written it, we make the software inherently high quality. One way to do this is minimise the opportunity to make a mistake and introduce a bug. If we make it easier to do the right thing, we’re less likely to make a mistake. Sometimes this is referred to as setting ourselves up for success.

That principle underlies the reasons that I prefer the lambda passing style. It’s impossible to access the database connection and forget to close it, or to access a stale connection, so we should never be able cause a bug that way.

CruiseControl.NET Campfire plugin

2010-01-15T09:01:00.001Z

On my current project, we’re using 37Signals’ Campfire for team chat and collaboration. We also use CruiseControl.NET for continuous integration. I thought it would be nice to have CruiseControl.NET automatically post build results into the Campfire chat room, to raise visibility of the build status out of the system tray and into the browser.

Implementing CruiseControl.NET plugins is quite straightforward, and Campfire has a very simple XML-over-HTTP API for posting messages into chatrooms. The code for the plugin is available on my github account. There are instructions there for using the plugin in your environment.

For the CruiseControl.NET integration, I found this blog post from Kris Selden very useful. The one thing I wasn’t sure about was how to handle errors. I didn’t want to risk exceptions being thrown from my plugin causing the build to fail. If something’s going to go wrong, I’d rather the plugin failed silently – we’ll quickly notice the lack of messages appearing in Campfire and look into it – so I made sure that I catch all exceptions and use CruiseControl.NET’s log4net facility to write to the server log.

I did have one problem with the Campfire API. I forgot to set the Content-Type HTTP header. Without this I was getting the HTTP response code 422 Unprocessable Entity, with a message “Body can't be blank”. This confused me for a while as I was definitely sending a well formed XML message with a body element. Setting the Content-Type header to application/xml fixed the problem.

Red-Green-Refactor at story level

2009-10-07T13:17:00.004+01:00

A quick thought that came up this morning. When working on a story, we have two sets of responsibilities:

to the acceptance criteria and business value
to the technical quality of the codebase

We often forget to attend to the codebase once we've finished attending to the business value. We may end up with an unfinished refactoring. We are often in such a rush to claim the points. It's important that we keep the codebase clean. Red-Green-Refactor and "Make it work, make it right" apply at the story level as well as the code level. Satisfy the criteria, then make sure you're happy with the code quality.

NHibernate Bidirectional Cascade – when is it appropriate?

2009-09-04T09:33:00.001+01:00

I recently found a situation where I needed to use NHibernate’s cascade facility from both ends of a relationship. Normally, I’d use it only in one direction, but in this scenario the only practical solution was to use it in both directions. I’m unsure whether this is a good thing to do, so in this post I’ll:

talk about why I normally use it in one direction
talk about the situation that led to the bidirectional cascade
ask whether this was the right solution.

One-directional cascades

Normally, the cascade is tied to the concept of ownership in the model. We cascade changes from the parent (e.g. an Order) to its logical children (e.g. OrderLines). This fits in with the Domain-Driven Design concept of an aggregate root. If we have the appropriate cascades set up from the aggregate root to the contained entities, then we can follow a simple workflow:

load up an aggregate root from the repository
modify, add and remove entities within the aggregate
let NHibernate automatically persist the changes to the database when the session is flushed

This makes for very clean domain layer code which remains persistent ignorant. The domain code that is performing these modifications doesn’t have to explicitly save or delete any of the entities within the aggregate.

I suppose this model does not say that we can’t have cascade running from children to parents. But I always felt that the natural order of things was to have cascades running in one direction.

Why we needed bidirectional cascades

In this scenario, the Customer is an aggregate root containing the ShoppingOrder and OrderLine objects. I’ve set up cascades from the Customer down to the OrderLine class, which allows the following code:

using (new SessionScope())
{
    var customer = ActiveRecordMediator<Customer>.FindByPrimaryKey(customerId);
    var shoppingOrder = customer.AddOrder(some, parameters);
    var newOrderLine = shoppingOrder.AddLine(some, parameters);
}

which executes SQL like this:

SELECT ... FROM Customer WHERE Id = ...
INSERT INTO ShoppingOrder ...
INSERT INTO OrderLine ...

Perfect. Note that I’m using Castle.ActiveRecord to configure and access NHibernate, with auto-flushing sessions. As far as I know, this scenario would translate directly to an equivalent plain NHibernate set up.

The problem comes with the OrderLineState. In our application, we have a separate part of the system which tracks the state of orders. There are two implications of this being a separate concern:

The OrderLineState is not considered part of the Customer aggregate. There’s no relation from the OrderLine to the OrderLineState.
The piece of code that creates the initial state object for an OrderLine is logically a long way from the piece of code that creates the OrderLine.

This means that when we create the OrderLineState, we need to explicitly tell NHibernate about the new entity:

using (new SessionScope())
{
    var customer = ActiveRecordMediator<Customer>.FindByPrimaryKey(customerId);
    var shoppingOrder = customer.AddOrder(some, parameters);
    var newOrderLine = shoppingOrder.AddLine(some, parameters);

    // In some other part of the system, perhaps responding to a domain event
    var state = new OrderLineState(newOrderLine);
    ActiveRecordMediator<OrderLineState>.Save(state);
}

But this code throws an exception from the call to Save(state):

NHibernate.PropertyValueException: not-null property references a null or transient valueNHOddities.OrderLine.Parent

This makes sense. We’ve asked NHibernate to save the OrderLineState object. This has a non-null, cascading reference to the OrderLine, so it tries to save the OrderLine. Because the relationship OrderLine.Parent (the Order) does not have cascading configured, NHibernate won’t try to save the Order yet. So the Order referenced by OrderLine.Parent is a transient object, and we get the corresponding exception.

There are a variety of ways to fix this. All of them work, in that they allow the code above to run and make the correct INSERT statements in the database.

Remove the NHibernate NotNull constraint on the OrderLine.Parent property. This lets NHibernate save the OrderLine before it’s saved the order. I don’t like this for two reasons. Firstly, the NotNull constraint reflects a genuine constraint: an OrderLine must always belong to an order. Secondly, that NotNull constraint is probably reflected on the database column definition, so we’d have to change the database schema, removing a meaningful and useful constraint. In general, I don’t like having to change what is a genuine reflection of the domain just to make things convenient for the framework I’m using.
Explicitly flush the session after adding the Order and OrderLine to the Customer aggregate, and before saving the new OrderLineState. I don’t like this one either. It requires my domain code to explicitly deal with session management. We usually consider this to be an infrastructural concern. We have a web app, so we handle session and transaction management at the request level.
Add a cascade on the BelongsTo relation OrderLine.Parent.

Introducing this bidirectional cascade seems like the best (or at least, least-worst) solution. But it doesn’t quite seem right.

Is there anything wrong with a bidirectional cascade?

As I discussed earlier, it feels like cascades should run from parents to children, not the other way, and not in both directions. This isn’t written down as a rule or guidance anywhere that I can find in the NHibernate documentation, but it seems to be the standard approach. Indeed when I tweeted about this, I got a reply from none other than one of the authors of NHibernate In Action himself:

@ascordellis Cascade on both sides feels heavy, are you sure that both sides need to be the big daddy of the relationship?

For now, I’m leaving it in. For our problem, it’s the least intrusive solution. But I’d love to hear your opinion. Is there a better way to solve our problem? Should NHibernate just deal with it for us?

How to recover lost commits in a git repository

2009-07-21T21:01:00.001+01:00

Today we had a problem: we lost several git commits when a rebase went wrong. We’re still not sure how this happened, but we were able to recover the commits. Here’s a quick summary of how we managed that, which I hope will help others get out of similar situations.

We tried to abort a rebase that had a lot of conflicts, and it took us from a situation like this:

to one like this:

Disaster! Our “last big commit on my story” has vanished! We were pretty sure it was still in the repository somewhere, it just wan't reachable from any named branch. We hunted around in git log, git fsck and other commands with no luck. It turns out that the answer lies in git reflog show

$ git reflog show
3330b4e HEAD@{0}: checkout: moving from ms2 to my-story
3330b4e HEAD@{1}: checkout: moving from 3330b4e0b9af502908ce2c59958ec1e7f83a1a4b to ms2
3330b4e HEAD@{2}: checkout: moving from my-story to 3330b4e
5b50352 HEAD@{3}: rebase: updating HEAD
8f4a119 HEAD@{4}: rebase: more work on my story
c715f09 HEAD@{5}: rebase: some work on my story
6e1441d HEAD@{6}: checkout: moving from my-story to 6e1441d03b200d62f9ffc28d902c60a289d2ea3b^0
5b50352 HEAD@{7}: checkout: moving from mainline to my-story
6e1441d HEAD@{8}: commit: more work by the rest of the team
e72f33f HEAD@{9}: checkout: moving from my-story to mainline
5b50352 HEAD@{10}: commit: last big commit on my story
3330b4e HEAD@{11}: commit: more work on my story
a7d49b0 HEAD@{12}: commit: some work on my story
f3d00fb HEAD@{13}: checkout: moving from master to my-story
f3d00fb HEAD@{14}: checkout: moving from mainline to master
e72f33f HEAD@{15}: commit: some work done by other developers
f3d00fb HEAD@{16}: checkout: moving from master to mainline
f3d00fb HEAD@{17}: commit: commit b
2fcece2 HEAD@{18}: commit (initial): commit a

There it is! We now have the SHA for the “last big commit on my story”. We can check that this really is the right one:

$ git show 5b50352

commit 5b503520f132e1a64c510816fcd61cc97626791e
Author: Alex Scordellis 
Date:   Tue Jul 21 20:26:21 2009 +0100

    last big commit on my story

diff --git a/anotherfile.c b/anotherfile.c

  ...diff contents elided...

Now all we had to do was reattach a branch name to that SHA, and we were back where we started:

ascordel@ALEXSCORDEL /c/alex/code/git-recovery
$ git checkout 5b50352
Note: moving to '5b50352' which isn't a local branch
If you want to create a new branch from this checkout, you may do so
(now or later) by using -b with the checkout command again. Example:
  git checkout -b 
HEAD is now at 5b50352... last big commit on my story

ascordel@ALEXSCORDEL /c/alex/code/git-recovery (5b50352...)
$ git checkout -b recovered-my-story
Switched to a new branch 'recovered-my-story'

Now we’re back where we were at the beginning. We can start again and try to get the rebase through this time. We still don’t know where it went wrong in the first place, but we were pretty relieved to get our work back after a short period of panic!

Problems mixing functional and object-oriented paradigms: NHibernate

2009-03-25T16:38:00.001Z

Imagine a typical operation being performed in a regular CRUD business app. In most cases, this will be written in an object-oriented language and will involve mutable objects.

grab some objects from the DB
modify their state according to some business rules
save the new state back to the DB

This pattern, in which an object changes state over time to represent changes in the ‘real world’ that we are modelling, is probably the most prevalent model found in business software systems which use an object-oriented language. Let’s call it the mutable-object pattern.

On the other hand, if we were to write this operation in a functional programming language, we are likely to do things differently:

take some data structures as input to the function
compute the values which result from applying some operation to that data
return a new set of data structures representing those results

Crucially, this code written in the functional style does not alter, or mutate, the data values in the input data structures. These structures are typically immutable.

There’s been some debate recently about how, when and if we can, or even should, mix these two styles. The addition of LINQ to C# has brought a functional paradigm (delayed execution of queries over collections) to an OO language in which most developers are used to the mutable-object pattern, and really brought this issue into the mainstream.

I think that it’s important to represent each part of your code in the way which most clearly expresses your intention. If your intention is to modify some existing data in a relatively simple way according to some rules, the mutable-object pattern works well. If you need to operate on collections of data structures, performing operations which are more algorithmic in nature, a functional style works well. For example, a SQL query expresses very clearly, using a functional paradigm, what you are trying to achieve. What would you rather read? SELECT MAX(Age) FROM People; or:

max = 0;
foreach (var person in people)
    if(person.Age > max)
        max = person.Age;

So there is benefit from mixing the mutable-object and functional styles, but we need to be careful at the boundary between areas of code written in these different styles. I came across a significant example of this recently. NHibernate assumes that you will use the mutable-object style I described at the top of the post. In particular, it expects that there is only one copy in memory of each entity represented by a database row. If it returns you a CLR object in response to a query, it doesn’t let you save a different CLR object back to the same database row. If you want to modify the data in the database, you need to modify the CLR object that NHibernate gave you.

We shouldn’t really be surprised by this. The boundary between object-oriented programming and relational data storage has long provided thorny problems for software development, because of the natural impedance mismatch between OO programming and relational data (ironically, this is exactly the problem that NHibernate does a pretty good job of solving). There’s a similar impedance mismatch between the mutable-object pattern and functional styles involving immutable data structures.

Footnote

I originally wrote this post using the phrase “the OO paradigm” to describe the mutable-object pattern, until Chris Leishman pointed out that using an OO language does not necessarily imply mutability. Mutability is just the typical way that we work with objects in an imperative language like C#. Similarly, I am not asserting that all functional programming involves algorithms consuming immutable objects and churning out new immutable objects, just highlighting one pattern typically found in functional programming.

ASP.NET doesn’t fire PostRequestHandlerExecute when you Response.Redirect

2009-03-21T12:34:00.001Z

The problem

We recently discovered that our application was leaking database transactions. Checking the logs revealed that we were not closing transactions whenever we redirect within the application.

The cause

We had been closing transactions in the event handler for the PostRequestHandlerExecute event exposed by HttpApplication. It turns out that this event is not fired when you call Response.Redirect(newUrl).

The solution

We moved our transaction closing code to the EndRequest handler. This event does get called, even when you redirect. You can also change your call to Response.Redirect(newUrl, false);, which causes ASP.NET to execute all the usual events in the HTTP pipeline.

The post-mortem

The MSDN documentation for HttpApplication gives you an ordered list of the events that are fired during a request, but does not tell you that you can disrupt the events with a Redirect. The MSDN documentation for Response.Redirect does tell you that you can control “whether execution of the current page should terminate”, but doesn’t tell you what that means. It’s not clear which events will and will not still fire.

We are using the Castle.Windsor/Castle.ActiveRecord/NHibernate stack, so we based our application class on the UnitOfWorkApplication in Ayende’s Rhino.Commons library. It uses the PostRequestHandlerExecute event to close the unit of work after handling a request. A unit of work isn’t quite the same thing as a transaction, so it may be deliberate that the unit of work is not closed on redirect. I’ve posted a message to the Rhino Dev Google group to find out.

My day at #QCon

2009-03-17T13:39:00.001Z

I spent last Wednesday at QCon. It was definitely an interesting day, but I felt like I missed a lot of stuff on other tracks. I also bumped into a few ex-colleagues from my pre-TW days and an old mate from uni who I hadn’t seen in years – I guess enterprise software is a small world…

First up was Sir Tony Hoare. Not having a background in computer science, I didn’t have quite the same feeling of awe as half the rest of the room. He made a few comparisons between software engineering (pragmatic, seeking dependability, short term view) vs computer science (perfectionist, seeking correctness, long term view), which were largely uncontroversial. Things got a bit interesting towards the end, where he stated his vision: “One day, software will be the most reliable component of every product which contains it … because of the successful interplay of research into the science of programming and the engineering of software”, and then fielded some interesting questions. It emerged that he sees complete specification as the limit of testing. I came away thinking that it’s important to have the computer scientists researching new ways for us to be certain about the behaviour of computer software, and to have the software engineers deciding pragmatically which of those methods are useful, and in what doses. Whether this will ever get us to Sir Tony’s vision, I’m not sure.

I popped into Ola Bini’s introduction to the emerging languages track, to see him explain why languages are important. My highlight here was Ola’s point that it’s all about communication, and we need to consider the best medium (i.e. language) for communicating each part of our systems to the machine, and to other humans. Since the target of our communication changes, so should the medium, and this is why we need different programming languages.

Next up, ThoughtWorks’ Ian Robinson on how to start an SOA initiative at enterprise scale. This was my pick of the day. It was one of those talks that makes you consider a whole new approach to an area of your work. I’ve picked out some key messages.

We should focus on capabilities which realise outcomes which deliver benefits (the what), rather than on services for services’ sake (the how). This allows us to focus on delivering those capabilities which differentiate our business from our competitors.
There’s a key difference between who is responsible for a business capability, and who is responsible for keeping a web service up. Engaging those in the first group will help engender a sense of ownership and urgency and ultimately lead us to more value-driven systems. For example, Ian suggests Responsibility Focused Stories (as <role>, I am responsible for <outcome>, which delivers <benefit>) to help identify these people and their responsibilities. From my own recent experience, I can attest that getting these business capability owners involved is often a significant challenge, particularly inside large companies, and I hope to use Ian’s ideas to help me do this.
Different service implementations are needed for different business capabilities, even if they operate on the same data. Don’t always assume that your business entities will take the same form and live in the same services. In particular the way we use and represent an entity may change significantly over the lifetime of that entity (consider the different capabilities involved in signing up a new customer, and performing daily financial trades on behalf of that customer). Update: see comment from Ian below for a clarification.
Use consumer-driven contracts to reduce coupling between service providers and consumers.
Have long-lived capability delivery teams responsible for business capabilities, but augment them with additional resources and knowledge from new projects that are taking advantage of the capabilities.

After lunch I was back in Ola’s track for Jonas Bonér on real-world Scala. He showed a simple chat application using the Lift framework, and whistled through a few of Scala’s more unusual constructs. Overall, I wasn’t convinced by what would make me choose Scala over any other functional programming language – it largely resembled Java with a few extra constructs thrown in. I think Jonas missed an opportunity to demonstrate the practical benefits that these constructs could bring and so get people excited it.

I then decided to go to Michael Feathers and Steve Freeman talk about the history of TDD. Having written and co-written Working Effectively with Legacy Code and Mock Roles, Not Objects respectively, I have a lot of respect for these guys. Those texts played key roles in my understanding of good OO design. The session was an interesting history lesson, a useful reminder of the core principles, and contained a few nice anecdotes. I was particularly tickled by the fact that early practitioners of testing (Gerry Weinberg, Kent Beck) just assumed that everyone else was doing testing too, as they considered it to be part of doing a professional job. Plus ça change.

Martin Fowler then described our experiences with Ruby projects at ThoughtWorks over the last few years. It was good to see some statistics backing up the idea that Ruby works in the enterprise. 36 of the 41 projects ThoughtWorks has undertaken in Ruby would use it again, and of those 5 that wouldn’t, 4 said that the reason was due to the problems getting people to adapt to Ruby, rather than Ruby’s ability to perform the job required. An amusing anecdote midway through: developers on a Ruby project estimated their productivity boost (over, say, Java) as 2-5x in earlier parts of the project, reducing to 1.5x in later parts, but jumping back up to 2-3x when they had to go back to Java and remembered just how bad it was! Martin’s a great speaker, so as always it was an entertaining session.

In conclusion, I enjoyed all the sessions and heard some interesting things. But the biggest lesson I’ll take away is to go to talks about things I don’t know about, not things I do. I’m sure I’d have learnt a lot more new ideas in Rich Hickey’s Clojure presentation than in Michael and Steve’s TDD history. Not through any fault of Michael and Steve (and I certainly don’t want to pick on them), but because I’ve read a lot of that stuff before. There were also a number of talks on web architecture and RESTful services that I’d like to have caught. I also wish I’d taken notes in more of the sessions; as well as helping me take more of the information on board it would have helped me writing this post!

For those who couldn’t make it (and others like me who want to catch up on sessions they missed), all of the slides are on the QCon site, although some may need an attendee-only password.

LINQ with side effects: convenient but dangerous

2009-01-09T17:56:00.001Z

Consider the following C# code. It’s trying to make sure that a Record exists for a given Thing, either by finding an existing Record or by creating a new one.

        public Record FindOrCreateRecordFor(Thing thing)
        {
            var recordFinders = new Func<Thing, Record>[]
                {
                    thing => retrieveExistingRecordFor(thing),
                    thing => createNewRecordFor(thing)
                };
            return recordFinders.Select(f => f(thing)).First(record => record.Exists);
        }

What happens if retrieveExistingRecordFor(thing) returns a record which has Exists == true? The answer, thanks to deferred query execution, is that createNewRecordFor(thing) is not called.

This is probably important to the correct functioning of our system; we don’t want to create a second, duplicate, Record for any one Thing. Because it’s important, it should be obvious from the code, and I’d argue that it probably isn’t. To someone used to imperative programming styles (which covers most C# programmers, certainly including me), it looks at first glance like both retrieveExistingRecordFor(thing) and createNewRecordFor(thing) will be called, then we’ll return the first result that has Exists == true.

[ By the way, if you don’t understand why createRecordFor(thing) is not called, you should read Jon Skeet’s Human LINQ post. For maximum effect, you should go to one of his talks and see it live, but reading about it is the next best thing. ]

What alternatives do we have in C#? We could write out the chain of alternatives explicitly:

        public Record FindOrCreateRecordFor(Thing thing)
        {
            var record = retrieveKnownRecordFor(thing);
            if (!record.Exists)
                record = createNewRecordFor(thing);
            return record;
        }

That's not too bad when there are two alternatives. In the example from my project that inspired this blog post, there were three, and I didn’t want to write a three-deep nested if. It also hardcodes the sequence of finder objects, which prevents you from doing any runtime configuration.

We could use an explicit foreach loop:

             foreach (var recordFinder in recordFinders)
             {
                 var candidateRecord = recordFinder(thing);
                 if (candidateRecord.Exists)
                     return candidateRecord;
             }

This is probably the best option, and the one I’ve ended up taking in our codebase. But I’m still not completely happy. I feel that there should be a better way. I can’t believe that you would write that code in a functional language, or even if you were writing in Ruby with a functional mindset. So if you are familiar with functional coding techniques, I have questions for you: How would you do this? Is it obvious to a functional programmer that this code is right?

    return recordFinders.Select(f => f(thing)).First(record => record.Exists);

Don't always use a mock

2008-12-23T17:34:00.001Z

It's easy to get carried away with mocking, especially when we have great frameworks like Rhino Mocks and Mockito helping us. I've definitely been guilty of it in the past, and I almost fell down that trap again last week. I had a method which needed to do three separate pieces of work in three separate transactions. It looked a bit like this:

    void Synchronise()
    {
        using (transactionService.LocalTransaction())
        {
            otherObject.OperationA();
        }

        using (transactionService.LocalTransaction())
        {
            otherObject.OperationB();
        }

        using (transactionService.LocalTransaction())
        {
            otherObject.OperationC();
        }
    }

I wanted to verify that each of the three operations occurred in separate transactions. It took me half an hour of frustration trying to accomplish this with Rhino.Mocks. I couldn't work out how to return three different transaction objects from the mock transactionService, and verify which of them was open during each mock call to otherObject. Then I took a step back, remembered that we don't have to use mocks for everything, and wrote the following classes:

public class FakeTransaction : IDisposable
{
    public bool InTransaction = true;

    public void Dispose()
    {
        InTransaction = false;
    }
}

public class FakeTransactionService : ITransactionService
{
    public readonly List<FakeTransaction> Transactions = new List<FakeTransaction>();

    public IDisposable LocalTransaction()
    {
        Transactions.Add(new FakeTransaction());
        return Transactions.Last();
    }
}

Now I can write tests like this:

otherObject
	.Stub(t => t.OperationA())
	.WhenCalled(invocation =>
	{
		Assert.That(transactionService.Transactions.Count(), Is.EqualTo(1));
		Assert.That(transactionService.Transactions.First().InTransaction, Is.True);
	});
	
otherObject
	.Stub(t => t.OperationB())
	.WhenCalled(invocation =>
	{
		Assert.That(transactionService.Transactions.Count(), Is.EqualTo(2));
		Assert.That(transactionService.Transactions.First().InTransaction, Is.False);
		Assert.That(transactionService.Transactions.ElementAt(1).InTransaction, Is.True);
	});

Much better! Especially when I added some nice helper methods to FakeTransactionService to do all the asserts in one line:

WhenCalled(invocation => Assert.That(transactionService.IsInTransaction(2)))

Again, this isn't a criticism of Rhino Mocks, which is another excellent tool, but a reminder that we shouldn't get bound to a particular tool, and always keep an open mind to alternatives. In general, this also echoes the idea that we should focus on the problem (verifying that each stage occurred in a separate transaction) rather than the solution (how to accomplish that in Rhino Mocks).

Using SQLite for fast database testing – technical gotchas and wider lessons learned

2008-12-23T11:58:00.001Z

On my current project, we’re successfully using SQLite for in-memory testing with NHibernate / Castle.ActiveRecord. This makes tests that run against the database a lot faster than if we ran them against SQL Server.

NHibernate’s pluggable driver / connection provider model makes this fairly painless, but there are a few gotchas in the technical details that are worth calling out. A couple of these led me to some general observations, neither of which are new but both of which are important.

Database lifetimes

An in-memory SQLite database lives for the lifetime of the connection, so each time you close and re-open the database, you lose your table structure and your data. Usually NHibernate opens and closes connections as it sees fit so you need to provide a custom DriverConnectionProvider for NHibernate which holds the connection open until you explicitly close it. This has been covered pretty well by other bloggers so I won’t go into details here. Just don’t forget to explicitly close the connection otherwise you’ll keep lots of databases hanging around taking up your memory.

The SqlType argument of the Property attribute

When we need to override NHibernate’s default choice of data type for a column, we can use the SqlType argument like so:

        [Property(ColumnType = "BinaryBlob", SqlType = "varbinary(max)")]
        public byte[] Data { get; set; }

There’s a problem here – varbinary(max) is a SQL Server column type that isn’t valid in SQLite. SQLite has the BLOB column type for binary data like this. We’d like to be able to pick a different SqlType for production (SQL Server) and test (SQLite) builds. However because SqlType is in an attribute declaration, it has to be a compile time constant. I found that the simplest way around this was to hook into the ActiveRecord initialization pipeline to modify the ActiveRecord metadata before the schema gets created.

        [TestFixtureSetUp]
        public virtual void TestFixtureSetUp()
        {
            ActiveRecordStarter.ResetInitializationFlag();
            ActiveRecordStarter.ModelCreated += ActiveRecordStarter_ModelCreated;
            ActiveRecordStarter.Initialize(typeof (MyEntity).Assembly, ActiveRecordSectionHandler.Instance);
        }

        private static void ActiveRecordStarter_ModelCreated(ActiveRecordModel model, IConfigurationSource source)
        {
            if (source.GetConfiguration(typeof (ActiveRecordBase))
                .Children
                .Single(configuration => configuration.Name == "connection.driver_class")
                .Value.ToLower().Contains("sqlite"))
                foreach (var property in model.Properties)
                    if (!string.IsNullOrEmpty(property.PropertyAtt.SqlType) && property.PropertyAtt.SqlType.ToLower().Contains("varbinary"))
                        property.PropertyAtt.SqlType = "BLOB";
        }

Column types and the law of leaky abstractions

SQLite and SQL Server offer different sets of column types. For example, SQLite has a TEXT column type for all textual data, whereas SQL Server has fixed and variable length text datatypes like NCHAR(100) and NVARCHAR(MAX). NHibernate and ActiveRecord hide these differences from you. You provide a property declaration and it generates the appropriate DDL for the database which you’ve configured NHibernate to use. For example, here are excerpts from the creation scripts for my current project.

For SQLite

create table ErrorLogs (
  Id INTEGER not null,
  RequestUrl TEXT,
  Message TEXT,
  primary key (Id)
)

For SQL Server

create table ErrorLogs (
  Id INTEGER not null,
  RequestUrl NVARCHAR(255) null,
  Message NVARCHAR(255) null,
  primary key (Id)
)

Notice the different datatypes and nullability declarations between the two databases. This starts to matter when you write a test that you can insert a really long (> 255 characters) string into the Message column. Your test passes fine against SQLite, then you run your app against SQL Server and it falls over horribly.

We can fix this example by making individual tests run against SQL Server, then override the default column types in the ActiveRecord attributes to make them pass.

        [Test]
        public void should_save_error_log_with_long_message()
        {
            using (SqlServer)
            {
                var moreThan255Chars =
                    "a really really long message a really really long message a really really long message a really really long message a really really long message a really really long message a really really long message a really really long message a really really long message a really really long message ";
                var savedErrorLog = new ErrorLog
                                        {
                                            Message = moreThan255Chars
                                        };

                systemUnderTest.Save(savedErrorLog);

                Flush();

                var retrievedErrorLog = systemUnderTest.Load(savedErrorLog.Id);

                Assert.That(retrievedErrorLog.Message, Is.EqualTo(moreThan255Chars));
            }
        }

        // In a base class of your test fixture class
        protected IDisposable SqlServer
        {
            get
            {
                // Reconfigure NH to use SQLServer
                // Rebuild empty schema
                // DisposableAction invokes the delegate you give 
                // it when it's disposed (inspired by Rhino.Commons)
                return new DisposableAction(TestFixtureSetUp);
            }
        }

        // In the affected ActiveRecord class
        [Property(ColumnType = "StringClob", SqlType = "NTEXT")]
        public string Message { get; set; }

But this is another example of a situation where we have to remember each time we need to test a field like this, and if we forget, we may well introduce a bug. As I said yesterday, it’s much better to make it easy to do the right thing, so we should make sure that our automated build will expose significant differences between SQL Server and SQLite automatically. Unfortunately, it seems that the only real option is to run all the tests again against SQL Server. This undoes the reduction in build time that we gained by using SQLite. I think a compromise is to run SQLite tests on the developers’ desktops and run SQL Server tests on the build server.

This scenario serves up a couple of interesting lessons:

Lesson 1: The Law of Leaky Abstractions

NHibernate would like you to think that it’s successfully abstracted away the differences in column types and SQL Syntax between the SQL Server and SQLite, but sometimes you need to know what’s behind the façade. This is a classic example of Joel’s Law of Leaky Abstractions. NHibernate and ActiveRecord have several leaks in their abstractions, as recently my colleague Stuart recently pointed out. They make you feel that you don’t need to worry about the presence of a relational database, but every now and then you need to remember that there is some SQL happening under the hood – in particular around session management and lazy-loading. This isn’t a criticism of ActiveRecord – it’s a great tool and they do explicitly call out some of these leaks on the project homepage.

Lesson 2: Every difference between your test and production environments will trip you up at some point

I think the headline speaks for itself. In this example, it was the difference between SQL Server and SQLite. This can happen in many areas – it’s not only differences in the technical environment, but differences in datasets and usage patterns can hit you too:

Testing and developing against tiny datasets, so you don’t notice the fact that your app is dog slow with a realistic dataset.
Forgetting that, unlike in development, the users’ browsers aren’t on the same machine, or even the same continent, as the web server, so maybe that really chatty AJAX UI isn’t the best plan.
Viewing the UI on the 1920*1080 mega-displays that the developers have, and forgetting that the user with a 1024*768 can’t see the navigation links without scrolling sideways.
Testing and developing against tiny datasets, so you don’t realise that you haven’t offered the users any easy way to find their data. Carlos puts it better than I can:

Next time you see a system where there are enormous listings of items with no search, pagination or sorting, ask the developers if they have ever watched a typical user at work; chances are they have only thought about the system as they see it: since the testing dataset is usually small, a loop spitting out a bit of HTML for each element isn’t such a big deal. They might even say there’s a story to implement all that lovely stuff later on, but they just get moved over and over to the bottom of the backlog barrel… until everyone watches a person struggle to find needles in a tabular haystack all day. This is a simple example – almost too trivial actually, but one I’ve seen happen way too many times.

What do we do about this? On the technical side, we can work to make our development and testing environments as close to production as possible. For the other areas, it’s vital that we are vigilant and keep an eye out for these risks as we go along. To paraphrase Thomas Jefferson: “the price of Agile (not having to write stacks of functional and non-functional specifications at the start of the project) is eternal vigilance (having to keep an eye on all these things as we go along)”.

HTML escaping in ASP.NET MVC

2008-12-22T15:37:00.002Z

HTML escaping is a key part of web site security, mainly to prevent Cross-Site Scripting (XSS) attacks. I’m no security expert, but I do know that everything the user types in must be HTML escaped before rendering it in the browser, otherwise bad people could do nasty things like write javascript which steals all the users cookies.

HTML escaping isn’t hard to do. You just convert < into < and a few other conversions. .NET provides a very simple way to do it:

    var encodedContent = HttpUtility.HtmlEncode(userEnteredContent);

ASP.NET MVC makes it pretty easy too:

<p>
    <%= Html.Encode(ViewData.Model.UserEnteredContent) %>
</p>

But the tricky bit is remembering to do it every single time. If we forget in once place in the whole app, we’ve introduced a security hole. It’s also pretty hard to test automatically that we’ve covered every possible instance – we could write a test for each part of the markup that we know contains user-entered content, but we’re just as likely to miss a test as to miss a call to Html.Encode(). Plus, it would be pretty boring and time-consuming.

I’m very much a fan of making it easier to do the right thing than to do the wrong thing. On my current project, we decided to look for ways to make HTML escaping happen automatically. This way, we’d be secure by default, and have to explicitly mark out those parts where we want to allow HTML in user content (e.g. if we had a facility where users could enter HTML formatted comments). We discussed a few options.

Encoding on the way in

We could choose to encode the user’s data on the way in, and store it in the database in escaped form. This could be implemented using a filter, or with some code in Global.asax.cs. However, this feels wrong, for a few reasons:

HTML encoding is a rendering problem, so the solution should be in rendering.
We aren’t storing the data as the user intended it, possibly reducing our options for using this data in the future (e.g. consumption by non-HTML fronted systems).
If you’ve encoded on the way in, you mustn’t then encode on the way out. Otherwise you will double encode and the user will see all sorts of HTML entities like > when they typed >. Many ASP.NET MVC helper controls (e.g. <%= Html.ActionLink(ViewData.SomethingTheUserEntered, "action", "controller") %>) escape the text you pass them, so you’d have to rewrite all the helpers without the encoding.

There’s a longer thread about the rights and wrongs of encoding on the way in on the MSDN forums. At the moment, I’m firmly in the wrong camp.

Encoding on the way out

Having decided not to encode on the way in, we considered two ways to make encoding happen by default.

Between the Controller and the View: Use a filter to encode all strings in the ViewData object passed from the controller to the view
In the View: Make the <%= %> syntax automagically do HTML escaping. Oh, but don’t do it when I’m generating tags, as in <%= Html.ActionLink(…) %>

Option 1 is fiddly and imperfect. I wrote a filter to examine the contents of the ViewData dictionary and the ViewData.Model, using reflection to search for read-write properties of type string, and replace their value with the encoded version. Not only is this very fiddly (think of all the recursive reflection), particularly when it comes to collections, but it can’t encode read-only properties or methods on the objects passed to the view. In hindsight, I shouldn’t be surprised that this is difficult – like encoding on the way in, this is a reponsibility being executed in the wrong place. This is a view concern and belongs in the view.

Option 2 isn’t trivial either – we need to hook into the ASP.NET compiler architecture to override <%= %> and need to find a way not to escape genuine HTML generated inside <%= %>. Fortunately we found Steve Sanderson’s blog post (good find Muhammad!) with a working solution.

Steve’s post is about a year old, which in ASP.NET MVC terms is positively ancient (Preview 2?), so it’s no longer the MVC toolkit that generates HTML, but the HtmlHelper class at the core of MVC. I didn’t fancy rewriting all of the MVC HTML generators (it was tedious enough going through the codebase removing Html.Encode(…)), so for now our views explicitly cast to Steve’s RawHtml class when necessary:

<span id="home">
    <%= (RawHtml) Html.ActionLink(ViewData.Model.SomeUserEnteredContent, "action", "controller") %>    
    <%= ViewData.Model.SomeOtherUserEnteredContent %>    
</span>

SomeUserEnteredContent will be encoded by Html.ActionLink, and SomeOtherUserEnteredContent will be encoded by <%= %>. The <a> tag emitted by Html.ActionLink will not be encoded.

Other Options

We also considered the possibility that HTML encoding could be a domain concern. In my opinion, this could be the case for an application that was heavy on user-entered marked-up text, such as a blog engine or an issue tracker where all the text fields are Textile formatted. Then each different field knows whether to allow HTML markup through, and if so, which HTML tags are considered safe, and this is part of the domain model of the application.

Conclusion

This seems like the most efficient way to make it easy to do the right thing. I’d be very interested to hear any other views, especially if you think this is not the right thing to do.

Finally, I’m surprised by how little coverage this has received within the ASP.NET MVC community and example code that you see out there on the web. Both that forum thread and Steve’s blog post date back a year. It is just assumed that everyone knows about this?

Using XmlSerializer to deserialize into a Nullable

2008-11-20T20:41:00.011Z

I recently had trouble deserializing an XML message which contained a nillable property which maps to a nullable value type in my C# class. I had the XML shown below (from a RESTful Rails service), and I wanted to map it to a class like this one:

    
    public class Entity
    {
        [XmlElement("id")]
        public int Id { get; set; }

        [XmlElement("parent-id")]
        public int? ParentId { get; set; }

        [XmlElement("name")]
        public string Name { get; set; }
    }

<?xml version="1.0" encoding="UTF-8"?>
<entities type="array">
  <entity>
    <id type="integer">1</id>
    <name>Foo</name>
    <parent-id type="integer" nil="true"/>
  </entity>
  <entity>
    <id type="integer">2</id>
    <name>Bar</name>
    <parent-id type="integer">1</parent-id>
  </entity>
</entities>

As you can see, the XML represents a set of objects with a parent-child relationship. Root elements don't have a parent, so their ParentId field should be null. However when I used the XmlSerializer to deserialize this, it threw me a "System.FormatException: Input string was not in a correct format.".

Internet searches didn't show me anything, so I resorted to writing my own NullableInt type to use in place of int?. I'd be surprised if XmlSerializer doesn't have a built in way to handle nullable value types. What am I missing?

    public struct NullableInt : IXmlSerializable
    {
        private int value;
        private bool hasValue;
 
        private NullableInt(int value)
        {
            hasValue = true;
            this.value = value;
        }
 
        public bool HasValue
        {
            get { return hasValue; }
        }
 
        public int Value
        {
            get { return value; }
        }
 
        XmlSchema IXmlSerializable.GetSchema()
        {
            return null;
        }
 
        void IXmlSerializable.ReadXml(XmlReader reader)
        {
            if (reader.GetAttribute("nil") == "true")
            {
                ReadNullValue();
                return;
            }
            ReadNonNullValue(reader);
        }
 
        void IXmlSerializable.WriteXml(XmlWriter writer)
        {
            throw new NotSupportedException();
        }
 
        private void ReadNullValue()
        {
            hasValue = false;
        }
 
        private void ReadNonNullValue(XmlReader reader)
        {
            reader.ReadStartElement();
            var s = reader.ReadString();
            value = Convert.ToInt32(s);
            reader.ReadEndElement();
            hasValue = true;
        }
 
        public static implicit operator NullableInt(int value)
        {
            return new NullableInt(value);
        }
    }

EDIT: See this StackOverflow post for the solution.

How to stop an XmlReader taking 5 seconds to load an HTML page

2008-10-30T20:42:00.010Z

If you've come for the answer to the title, skip to the end!

On my current project, we are writing a simple WebDriver inspired framework for acceptance testing our web app (we'd give WebDriver a go, but we're on .NET). The framework gets the pages from the server using HttpRequests, inspects them using XPath etc, then builds forms to post back to the server. We chose to do this to make our tests run faster than they would in Selenium (of course, when there's JavaScript involved, we have to resort to Selenium).

Unfortunately, we found that our tests were still taking a long time: 10-15 seconds to load a page, fill in a form, post it back and view the response. We had been hoping for much better performance - at this rate we might as well use Selenium! I was puzzled as to what was taking so long - surely it shouldn't take 5 seconds to load a smallish HTML page into an XmlDocument? I converted the code to use XPathDocument instead, as it's meant to be faster for read-only access, but saw little or no improvement (not really surprising on a small document).

It wasn't until I was packing up to go home, and unplugged my laptop from the network while it was running these tests, that I found the problem:

               [nunit2] 9) A.Test.Name : System.Net.WebException : The remote name could
not be resolved: 'www.w3.org'
               [nunit2]    at System.Net.HttpWebRequest.GetResponse()
               [nunit2]    at System.Xml.XmlDownloadManager.GetNonFileStream(Uri uri, ICredentials credentials)
               [nunit2]    at System.Xml.XmlDownloadManager.GetStream(Uri uri, ICredentials credentials)
               [nunit2]    at System.Xml.XmlUrlResolver.GetEntity(Uri absoluteUri, String role, Type ofObjectToReturn)
               [nunit2]    at System.Xml.XmlTextReaderImpl.OpenStream(Uri uri)
               [nunit2]    at System.Xml.XmlTextReaderImpl.DtdParserProxy_PushExternalSubset(String systemId, String publicId)
               [nunit2]    at System.Xml.XmlTextReaderImpl.DtdParserProxy.System.Xml.IDtdParserAdapter.PushExternalSubset(String systemId, String publ
icId)
               [nunit2]    at System.Xml.DtdParser.ParseExternalSubset()
               [nunit2]    at System.Xml.DtdParser.ParseInDocumentDtd(Boolean saveInternalSubset)
               [nunit2]    at System.Xml.DtdParser.Parse(Boolean saveInternalSubset)
               [nunit2]    at System.Xml.XmlTextReaderImpl.DtdParserProxy.Parse(Boolean saveInternalSubset)
               [nunit2]    at System.Xml.XmlTextReaderImpl.ParseDoctypeDecl()
               [nunit2]    at System.Xml.XmlTextReaderImpl.ParseDocumentContent()
               [nunit2]    at System.Xml.XmlTextReaderImpl.Read()
               [nunit2]    at System.Xml.XPath.XPathDocument.LoadFromReader(XmlReader reader, XmlSpace space)
               [nunit2]    at System.Xml.XPath.XPathDocument..ctor(Stream stream)

I was surprised to see the XmlTextReader going off to the web to fetch the DTD specified in my page's DOCTYPE tag. I suppose it is sometimes useful to validate pages in this way, but I wasn't really expecting it here.

After a bit of googling, I found the answer on the increasingly useful stackoverflow.com. Just set the XmlResolver on your XmlReader to null, something like this:

var reader = new XmlTextReader(lastResponse.GetResponseStream()) {XmlResolver = null};
var navigator = new XPathDocument(reader).CreateNavigator();

One lesson here is that it's a good idea to periodically run your build disconnected to the network, just to reveal any unexpected dependencies you may have. I guess this won't always be possible - you might have genuine dependencies - but it's a useful tool in the 'shake it and see what comes out' category.

Edit: It turned out that like Rasmus on stackoverflow.com I too needed to resolve entities. For example, my HTML contained a   entity. Without the DTD and entity definitions, I couldn't load my XML document. So I also had to write a LocalXmlResolver to return those DTDs I needed from a embedded resource. All it needs to do is recognise URLs like http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd and http://www.w3.org/TR/xhtml1/DTD/xhtml-lat1.ent and return the appropriate content.

Using Ruby to find non iTunes purchases to copy to a new machine

2008-09-19T11:54:00.008+01:00

I'm about to get a replacement laptop, so I need to copy all my stuff over, including music. With iTunes, it's easy to transfer songs that you've purchased from iTunes via your iPod (just don't forget to deauthorize your old machine before you dispose of it!), but not so easy to transfer all the music that you've downloaded from other sources, ripped from CDs, etc.

I wrote the following Ruby script to find all such files in my iTunes library and copy them to another folder, from where I can burn them to CD, transfer them over a network etc. It does rely on all your downloaded and imported files being MP3 encoded, rather than AAC, since it uses the file extension to determine the relevant files. It also excludes everything in the Podcasts directory, since I can always download them again.

Anyway, thought other people might find it useful.

require 'find'
require 'ftools'
dirs = ["c:/path/to/my/itunes/library"]
excluded_extensions = %w(m4p m4a m4b m4v pdf).map { |ext| '.' + ext }
excluded_dirs = %w(Podcasts)
target = "C:/music/to/copy/to/new/machine"
files = []
for dir in dirs
  Find.find(dir) do |path|
    if FileTest.directory?(path)
      if excluded_dirs.include?(File.basename(path))
        Find.prune
      else
        next
      end
    else
      if excluded_extensions.include?(File.extname(path))
        next
      else
        puts path
        files << path
      end
    end
  end
end

files.each do |file|
  File.copy(file, target + File.basename(file), true)
end

Converting Mercurial to Git on Windows

2008-09-11T11:55:00.005+01:00

Summary

If you want to convert a Mercurial repository to a Git repository, without losing history, on a Windows machine, use Stelian Pop's script, but you'll need to run it under Cygwin.
Prefer using msysGit for interactions with Github, since all the instructions for configuring it and dealing with newlines are clearly explained for you.

The Story

I've got a Ruby script for generating graphs of build duration and other metrics from CruiseControl, and it was in a Mercurial repository on my Windows machine. I'd chosen Mercurial for a couple of reasons:

I wanted to try out one of these newly popular distributed version control systems to see what the fuss is all about.
I was writing the code on my laptop, then copying it to a machine on my client's network so I could run it against our CruiseControl build. I wanted to be able to make modifications on the other machine, perhaps in parallel to modifications on my laptop, and port them back. With Mercurial, the process was simple (zip up my repository from my laptop, email it to myself at my gmail account, download onto client machine, unzip, modify code, commit, rezip, re-email, unzip on laptop in a different location, 'hg pull' from new location to original location).

Mercurial seemed relatively simple to use, coming from a Subversion background. You can use it locally just as if it was Subversion, then worry about pulling and pushing once you need the distributed facilities or need to share your code. It even has the TortoiseHg Windows shell integration.

However all the hype these days seems to be about Git. I also wanted to get my code up on the web so others could have a go, and Github seemed to be the most natural place to put it. I wanted to keep the history of the project, so I started looking for a way to convert a Mercurial repository to Git. I found Ben Hood's blog post. However my msysGit bash shell installation (where I had been doing most of my Git work), did not have Python available. So I installed the git package into my local Cygwin installation, and the script worked there. I then went back to msysGit's bash shell for the upload to my Github repository, since I'd configured it with my SSH key etc as suggested by the Github guide for Windows.

Generics and reflection in .NET

2008-09-02T16:56:00.012+01:00

This week I've been working on the implementation of generics in NSynthesis. It's mostly involved some pretty hairy reflection code, often with a lot of edge cases.

To verify that you have tests of all production implementations of mocked methods, NSynthesis goes through two main procedures

For each mock call observed, identify all possible implementing methods.
For each real call observed in a unit test, tick off an implementing method we identified in step one.

Both of these steps aren't too hard when there aren't generics involved. For step one, just walk the inheritance tree looking for methods with the appropriate signature. For step two, you can just ask the MethodInfo objects to compare each other.

Generics introduce a whole new set of problems. I posted last week about the rules that I think we should follow, but implementing them is pretty hard. For example, if we ask the CLR: typeof (IService<Customer>).IsAssignableFrom(typeof (Service<>)));, the answer is rightly false, but that makes it harder to detect that a method you've found on Service<T> is a potential implementation for your mock call to a method on IService<Customer>.

I made some progress on step one, to the extent that all the implementations of IService<Customer> that I mentioned in my earlier post are picked up. It took me a while and is nowhere near finished (e.g. it won't find implementations of mock calls that are not to interfaces) but I did learn a few useful things along the way:

Type.IsGenericTypeDefinition
Type.IsGenericType
MethodInfo.IsGenericMethodDefinition
MethodInfo.IsGenericMethod

IsGenericType will return true for any Type object where the type has type parameters, e.g. typeof(Repository<Customer>) or typeof(Repository<>).

IsGenericTypeDefinition will only return true for typeof(Repository<>), not typeof(Repository<Customer>). You can go between typeof(Repository<>) and typeof(Repository<Customer>) with the methods GetGenericTypeDefinition() and MakeGenericType(params Type[]). Note that to make a generic type from a generic type definition, you need to provide the type parameters.

The same thing applies to methods, with the one gotcha that IService<T>.Method() is not a generic method, and IService.Method<T>() is. In other words, you need the method itself to have the generic type parameters, rather than its declaring type.

Having got that far with step one, I decided to move to step 2: matching test-runtime invocations of production code to the methods that we decided needed testing during step one. The good news here is that inheritance doesn't get involved. We now just want to check that every method was called. It sounds easy, and prior to generics, it was. In fact, Stuart's implementation which worked for non-generics just compared the two MethodInfo objects using their Equals method. This won't work for generics - remember that we want Service<Widget>.Load() to satisfy our need for a test of Service<Customer>.Load(), but the MethodInfo's typeof(Service<Widget>).GetMethod("Load") and typeof(Service<Customer>).GetMethod("Load") are not counted as equal.

It turns out that what we get from PostSharp capturing our test call is equivalent to typeof(Service<Customer>).GetMethod("Load"), but what we got from step one of NSynthesis is equivalent to typeof(Service<>).GetMethod("Load"). I started digging deep into the reflection objects, with code like

        private bool MethodCalledIsDeclaredOnClosureOfGenericTypeDefinition(Type typeDeclaringMethodToSatisfy)
        {
            return
                typeDeclaringMethodToSatisfy.IsGenericTypeDefinition &&
                TypeDeclaringMethodCalled.IsGenericType &&
                !TypeDeclaringMethodCalled.IsGenericTypeDefinition && TypeDeclaringMethodCalled.GetGenericTypeDefinition().Equals(typeDeclaringMethodToSatisfy);
        }

and

        private bool SignatureMatchesGeneric(MethodInfo methodToSatisfy)
        {
            if (methodToSatisfy.Name != methodCalled.Name)
                return false;
            if (methodToSatisfy.GetParameters().Count() != methodCalled.GetParameters().Count())
                return false;
            for (int i = 0; i != methodToSatisfy.GetParameters().Count(); i++)
            {
                if (methodToSatisfy.GetParameters()[i].ParameterType.IsGenericParameter)
                {
                    var genericParameterPosition = methodToSatisfy.GetParameters()[i].ParameterType.GenericParameterPosition;
                    if (!(methodCalled.GetParameters()[i].ParameterType == methodCalled.DeclaringType.GetGenericArguments()[genericParameterPosition]))
                        return false;
                }
                else
                {
                    if (!methodToSatisfy.GetParameters()[i].ParameterType.Equals(methodCalled.GetParameters()[i].ParameterType))
                        return false;
                }
            }
            return true;
        }

which does all sorts of comparing of positions of generic type parameters and so on. It only covers a few cases too. For example it wouldn't work if a parameter to a method had type IList<T> - I was dreading having to dig arbitrarily deep into nesting of generic parameters - nor in the case when both the type and the method had generic type parameters.

Fortunately I then discovered the MetadataToken property of the MemberInfo class. It appears (so far!) to give a unique number for each method body, which is exactly what we need for NSynthesis. So all that code above (and there was more of it!) got replaced by

        public bool Satisfies(MethodInfo methodToSatisfy)
        {
            return methodCalled.MetadataToken.Equals(methodToSatisfy.MetadataToken);
        }

Synthesized testing and .NET generics

2008-08-29T12:41:00.013+01:00

I'm working with my colleage Stuart Caborn on NSynthesis, a .NET implementation of the theory of Synthesized Testing. If you've not heard about Synthesis, or Synthesized Testing, I'll give a quick introduction. In general, the idea is to enforce that whenever you've mocked out a method during a unit test of a class calling that method, you've also got a unit test for all concrete implementations of that method. This increases your confidence that your mock tests and your concrete tests join up, without making you write any of those pesky slow functional tests.

There's much more to be found on Stuart's blog, and that of George Malamidis who wrote the original Ruby implementation.

The problem we're having with NSynthesis is: what do we do about generics? This blog post is me thinking out loud about this problem. We're pretty clear that if you've mocked IService.Method(), then you need a unit test for ConcreteService.Method() and OtherConcreteService.Method(), where ConcreteService and OtherConcreteService are all the known classes implementing IService. But what if I've mocked IService<Customer>.Method() or IService.Method<Customer>()? On which methods should we now be demanding unit tests?

I think if I've mocked methods on IService<Customer> and there exists a class CustomerService : IService<Customer>, then NSynthesis should demand unit tests of the corresponding methods on CustomerService. But what if there isn't a CustomerService, but rather just Service<T> : IService<T>? My first reaction is to demand unit tests for methods on Service<Customer>.

But what happens then when I mock IService<Product>? Do I now need unit tests for methods on Service<Product>? If I follow this path then I'll have many duplicate tests of my Service<T> class. That's got to be wrong. So unit tests of Service<Customer> should satisfy mock calls to IService<Product>.

But now I realise that for Widgets, I need a special implementation of IService<Widget>. So I introduce WidgetService : Service<Widget>. WidgetService overrides some of the methods in Service<Widget>. I should probably demand unit tests of these methods too (assuming I have mock calls to these methods on IService<Widget>).

If you're interested in Synthesized Testing you can join the mailing list or check out the Java version. I wonder how they are handling generics?