Jef Claes

The open plan fallacy testimonials

2012-05-27T15:36:00.000+02:00

I wrote an article titled 'The open plan fallacy' just two weeks ago. Earlier this week a similar article was published by the New York Times. The content of that article wasn't particularly extraordinary, but the comments were. I waded through all of them on my daily commute, and it's really hard to find one in favor of open plan offices - people seem to be enraged.

I handpicked some of the most interesting ones.

Research by Emberson et.al. (Psychological Science, 2010) demonstrate both the impairment in performance for people forced to listen to half a phone conversation and the neurological impossibility of "tuning out" these overheard conversations. The office experience has been described as trying to drive a car full of people all talking on their cell phones.

It's preferable to have to make the effort to collaborate than the effort to focus on one's work amidst a distracting open workplace. My profession, architecture, does require a high degree of collaboration, but also a need for intense, individual concentration working on complicated 3D computer models and details.. and long hours being in "the zone". My firm provides private offices for all of its staff and a variety of collaborative and recreational environments.. particularly the latter, recognizing that just by hanging out together for part of the day, individuals from different project teams and pursuits "cross-pollinate", come up with imaginative ideas, influence and inspire eachother.

Cubicles were invented as a way to shove more people into a smaller space and save money for corporations. The entire meme about "improved workplace communications" was invented by consultants as a smokescreen. Period. End of story.

"For example, someone whose job requires intense concentration (e.g. computer programmer) needs absolute silence. Programmers who do not get this silence are likely to make mistakes."
I've been around a lot of programmers, among lots of other kinds of workers, and I'd say that 90% of the developers I saw were in a large room with others all around them. It's actually one of the least likely jobs to provide someone with an office, despite the fact that your diagnosis about what the job requires is absolutely right, in my opinion.
All it takes is a few months in the world of corporations to understand that Dilbert is really a documentary.

I currently work in an open plan office and absolutely hate it. Not having an office with a door that others need to knock on before disturbing me has led to non-stop disturbances all day long. Not to mention having to listen to nonstop chatter of those around me. It is an incredibly inefficient - and I may add, unprofessional - way to work. Any money saved on rent is surely made up for in lost productivity.

I've noticed that the person who decides on the open cube layout usually sits in an office.

Having participated in the design of office environments, where we used low cubicle walls or even no walls to support certain kinds of collaborative work, with the full involvement of the employees in that design, it was always distressing to see the 'open office' faddishly embraced by management everywhere, regardless of actual practices required for the work. Consultancies, as usual, led the way in yet another blind embrace of 'innovation'.
Why would individual 'entrepreneurs' - to take one example from this story - want or need to be able to see and hear each other whilst working? Are the activities in which they are engaged intimately linked, are the tasks often (and necessarily) performed conjointly, is their own working division of labour a concerted one? What do these entrepreneurs themselves think about such matters? (Well, as the story makes plain, they have very clear answers, voting with their headphones!) Anyway, these are the kinds of very practical questions about work activities and worker needs that should drive office design.

In my article, I advocated isolating teams instead of individuals, but most commenters seem to be heavy supporters of private offices - including walls and a door. I never experienced that in a professional environment, so I couldn't say if that would work for software development. Can you? Collaboration and communication between developers is a necessity. Fostering that just seems hard when everyone is in a separate place - definitely for young teams. While comparing office layouts, I wondered how my proposed solution would scale; how many people can you put in one team before you encounter the same undesired open plan side-effects?

Painless database logging with mongoDB

2012-05-20T17:18:00.000+02:00

While browsing the source code of the ELMAH mongoDB provider, I learned about a special type of collections: capped collections.

From the mongoDB documentation:

Capped collections are fixed sized collections that have a very high performance auto-FIFO age-out feature (age out is based on insertion order). In addition, capped collections automatically, with high performance, maintain insertion order for the documents in the collection; this is very powerful for certain use cases such as logging.

This is such a killer feature. Logging to the database can be extremely useful, but also rather expensive. Using this feature, you can turn on database logging without too many worries.

Insertion into a capped collection is ridiculously fast. To get an idea of how fast it really is, I did some measuring on my own humble machine. I managed to insert 10.000 small documents in less than 3.7 seconds. The headaches of tweaking buffer sizes and rolling asynchronous appenders seem to be miles away.

Something which religiously gets ignored until shit hits the fan, is log table maintenance. With a capped collection there is no need to set up a database job that periodically cleans the logging table. You just set a fixed size, and you're done. No more middle-of-the-night support calls when the logging table is eating up all the disk space.

Creating a capped collection with the C# driver can look like this.

var server = MongoServer.Create("mongodb://localhost/");
var db = server.GetDatabase("PlayGround");

var options = CollectionOptions
    .SetCapped(true)
    .SetMaxSize(5000)
    .SetMaxDocuments(100);

if (!db.CollectionExists("Log"))
    db.CreateCollection("Log", options);

Now that's easy sailing.

Expect to find me posting more on mongoDB in the coming months, I'm starting to really like this little database engine.

The open plan fallacy

2012-05-13T16:20:00.000+02:00

I haven't worked in a whole lot of places, somewhere around four, but every single one of them used an open plan to structure their workplace. From what I hear from others, it's the standard.

There are a few things to say about the advantages of an open office layout. They should stimulate communication, create more opportunities for observing and learning from others and be more cost-effective. I'm afraid it's the latter which is the biggest driver though.

In reality open plans really aren't all that great. The noise alone has proven to reduce productivity by one third. When I look around, I see plenty of signs that people have a hard time getting their job done: programmers buying $500 noise-cancelling headphones in an attempt to keep the environmental noise out, project managers camping in free meeting rooms trying to focus on their number wizardry or even whole teams occupying a meeting room days before a release.

What bothers me the most is overhearing other teams, those distractions amount to nothing at all. What's the value of me overhearing a discussion on some obscure Sharepoint problem? None. What are the benefits of listening to your team's Friday after lunch bullshitting? Nothing. Team dynamics differ, that's normal, but they shouldn't disturb others.

The extreme alternative would be private offices, but I'm pretty confident that's far from perfect as well.

What I suggest, is installing each team in their own fully isolated area, free from distractions caused by other teams. This would still do right to all the advantages of an open plan, while taking away some of its biggest bottlenecks. By installing each team in their own cocoon, you create an environment where overhearing conversations on the next feature is an added value, where you can quickly short-circuit discussions on an architectural decision or where you might even concentrate on a problem for 45 minutes straight. The only advantage which is partially gone is cost-effectiveness; there needs to be some flexible infrastructure in place, that makes it easy to swiftly adapt the environment when team compositions change.

Instead of always focusing on that short term budget win, we need to start paying attention to the big picture again: offices should enable teams, not sabotage them.

Is there anybody who genuinely loves open plan offices? If not, what alternative would you prefer?

Why I will always love RSS

2012-05-06T16:50:00.000+02:00

There has been a lot of noise in the tech community earlier this year about how RSS is supposedly having one foot in the grave. If that would be even remotely true, I hope it dies with its boots on. The herald would be browsers and social networking sites killing or hiding support for RSS. While that may be true, their motives shouldn't rig our opinions.

RSS has never worked out for the regular consumer, not directly anyways. So I get why browsers are dropping support for it, I am not even disappointed. Most popular social networks have enough traction by now so that they can safely start fencing their gardens with the purpose of bringing more money in. Also reasonable.

What startles me is when peers start advertising the death of RSS. The explanations I hear are somewhere along the lines of "Why use RSS? I count on the people I follow on Twitter to share links to good information", or similarly "I just check Hacker News a few times a day to read up on the latest news". I strongly believe the contemporary fetish of liking and sharing cheapens the way we consume our information. Don't get me wrong, I do see value in community driven content, but there's also a lot of dirt and sensationalism. Some days it feels like I'm reading the front page of a cheap tabloid.

I used to subscribe to a bunch of feeds, but today I'm pickier; I subscribe to authors who share things I care about and who inspire me in some way. By subscribing to their feed, I'm not missing a single update. This forces me to absorb it all, even though most of their content doesn't go viral. This liberates me from feeling the urge to be connected all the friggin' time, plus more importantly, there is a gold mine of wisdom to be found in the non-controversial content out there.

I love RSS. It just works, and enables me to learn so much, without having to be connected constantly or having to rely on others to tell me what to consume.

Subscribe to my RSS feed here.

My InfoQ article on HTML5 offline web applications

2012-05-02T20:09:00.002+02:00

After writing a few things on HTML5 offline web applications earlier this year, I got contacted by InfoQ to write an in-detail article on the subject for them. I hesitated at first, because I was afraid that it would feel too much like work, taking the fun out of my writing. Turns out it wasn't like that at all. The guys at infoQ were really relaxed to work with, asking interesting questions and giving useful feedback, without forcing me into a certain direction. A pleasant experience.

You can find the article here.

Some Servicelocator pattern stinks

2012-04-17T20:33:00.000+02:00

I have been working on a somewhat legacy codebase which makes use of the Servicelocator pattern. Although I always thought of Dependecy Injection to be the superior pattern, I was pleased to find some Inversion of Control implementation in there. Working with the codebase, I discovered first hand how easily, when used without caution and discipline, the Servicelocator pattern can introduce code rot.

I will walk you through some of the issues I have with the Servicelocator pattern, mostly looking at it from a test perspective. It's interesting how you can often quickly discover friction in a codebase by just looking at, or writing, tests.

The first thing that rubs me the wrong way is that by using the Servicelocator pattern, you make each class dependant on the servicelocator, degrading the purity of the models. Although this is a conceptual consideration, it considerably affects development.

What I really like about Dependency Injection is that you can look at the constructor (or properties) from a class and instantly know its dependencies. The Servicelocator pattern makes it too easy to hide them.

To demonstrate some of the beef I have with the Servicelocator pattern, I wrote a FireStation class which has one public method Alert. I chose an example which is not the de facto OrderService or ShoppingBasket implementation, but which still is small enough to grasp easily.

So let's have a look at this FireStation class. I know it's a ridiculously naïve implementation, but good enough as an example.

public class FireStation
{
    public void Alert(Incident incident)
    {
        SendPagerMessage();
        SendEmail();

        if (incident.Priority == IncidentPriority.High)            
            ActivateSiren();                        
    }

    private void SendPagerMessage()
    {
        ServiceLocator.Current.GetInstance<IPagingTerminal>().SendMessage();
    }

    private void SendEmail()
    {
        ServiceLocator.Current.GetInstance<IMailServer>().SendMailMessage();
    }

    private void ActivateSiren()
    {
        ServiceLocator.Current.GetInstance<IImmoticaServer>().ActivateSiren();
    }
}

There is one public Alert method, which takes an Incident instance, and uses three alarmation channels to alert firemen: paging, email and a siren.

So, let's imagine I wanted to test whether the siren is activated when there is a high priority incident. I would just start writing the test, to ultimately find out I have no idea what to assert.

[TestMethod()]        
public void Alert_activates_the_siren_when_the_priority_is_high()
{
    var fireStation = new FireStation();

    fireStation.Alert(new Incident(IncidentPriority.High));

    Assert.Inconclusive("How can I verify the siren is activated?");
}

Looking at the collapsed method definitions, I still don't know. There is no constructor, so the dependencies aren't resolved in there. Nothing left to do but inspect the Alert method content, and look at the ActiviateSiren method implementation. That's where I eventually find out I need to mock the IImmoticaServer interface to assert the interaction.

So I do that, by creating the mock, setting up the container and setting it as the provider for my servicelocator.

[TestMethod()]    
public void Alert_activates_the_siren_when_the_priority_is_high()
{            
    var immoticaServer = new Mock<IImmoticaServer>();

    var kernel = new StandardKernel();
    kernel.Bind<IImmoticaServer>().ToConstant(immoticaServer.Object);
    
    ServiceLocator.SetLocatorProvider(() => new NinjectServiceLocator(kernel));    

    new FireStation().Alert(new Incident(IncidentPriority.High));

    immoticaServer.Verify(i => i.ActivateSiren(), Times.Once());
}

Now I'm polluting my test with responsibilities it shouldn't really care about. I could probably move that into some infrastructure or the test setup, but still, I'm writing code that could be easily averted.

Anyways, I should now be able to verify the interaction with the IImotticaServer implementation. Boom, red test. You probably figured this one out already, but if I hadn't expanded the private methods, you would have had no idea that I needed two extra stubs to make the test pass; one for the IPagingTerminal dependency and one for the IMailServer dependency.

kernel.Bind<IPagingTerminal>().ToConstant(new Mock<IPagingTerminal>().Object);
kernel.Bind<IMailServer>().ToConstant(new Mock<IMailServer>().Object);

After binding these two dependencies the test finally turns up green.

This development experience was horrible. The Servicelocator pattern fails at making it easy for me to discover dependencies, leading to an interrupted test flow. Another harmful side-effect is that it also becomes harder for me to read code and understand the high level interactions between objects, without getting knees deep into the implementation details.

By hiding your dependencies, you also promote ignorance to a problematic amount of dependencies. I discovered three in this example, but as this class grows, more and more of them will be buried inside the method implementations.

To make it easier to avoid and spot these smells earlier on, without dropping the Servicelocator pattern as a whole, you could move service resolution to the constructor. This isn't a completely safe refactoring though. It's possible that some instantiations are expensive, and should be implemented to be lazy initialized.

There is one more annoyance I would like to spout in this post. The Servicelocator pattern doesn't encourage good OO. I'm not an OO purist, but seeing dozens of static classes resolving their dependencies through a servicelocator sends shivers down my spine. I would rather pick an Inversion of Control technique which basically makes it impossible to do this.

This post turned out to be longer than I expected it to be, and I even left some annoyances uncovered. I hope I succeeded in explaining my beef with the Servicelocator pattern. I trust that the general consensus will soon dictate that this pattern easily leads to marginal code, and that it actually is an anti-pattern which should be avoided if possible.

What is your experience with the Servicelocator pattern?

Visualizing the offline application cache update progress

2012-04-11T16:27:00.001+02:00

I wrote about using the HTML5 application cache earlier, mostly focusing on generating and serving the manifest file using ASP.NET MVC. I also bitched about how not one browser I know of gives an indication of the application cache update progress. Today I wanted to write something about how you can visualize the application cache update progress yourself.

The applicationCache API has several useful and rather straightforward events we can handle to inform the user of the update progress.

window.applicationCache.onchecking = function (e) {
    updateCacheStatus('Checking for a new version of the application.');
};
window.applicationCache.ondownloading = function (e) {
    updateCacheStatus('Downloading a new offline version of the application');
};
window.applicationCache.oncached = function (e) {
    updateCacheStatus('The application is available offline.');
};
window.applicationCache.onerror = function (e) {
    updateCacheStatus('Something went wrong while updating the offline version 
                        of the application. It will not be available offline.');
};
window.applicationCache.onupdateready = function (e) {
    window.applicationCache.swapCache();
    updateCacheStatus('The application was updated. Refresh for the changes to take place.');
};
window.applicationCache.onnoupdate = function (e) {
    updateCacheStatus('The application is also available offline.');
};
window.applicationCache.onobsolete = function (e) {
    updateCacheStatus('The application can not be updated, no manifest file was found.');
};

One event that is particularly helpful is the progress event. This event fires every time a resource is downloaded and contains three useful attributes we can use to display the download progress: lengthComputable, loaded and total. These attributes should be fairly self-descriptive, but here is the relevant snippet of the W3C specificiations.

For each cache host associated with an application cache in cache group, queue a post-load task to fire an event with the name progress, which does not bubble, which is cancelable, and which uses the ProgressEvent interface, at the ApplicationCache singleton of the cache host. The lengthComputable attribute must be set to true, the total attribute must be set to the number of files in file list, and the loaded attribute must be set to the number of number of files in file list that have been either downloaded or skipped so far. The default action of these events must be, if the user agent shows caching progress, the display of some sort of user interface indicating to the user that a file is being downloaded in preparation for updating the application.

To have a text that updates the percentage of downloaded resources on every download, I came up with this.

window.applicationCache.onprogress = function (e) {               
    var message = 'Downloading offline resources.. ';

    if (e.lengthComputable) {
        updateCacheStatus(message + Math.round(e.loaded / e.total * 100) + '%');
    } else {
        updateCacheStatus(message);
    };
};

These attributes seem to be implemented in WebKit browsers, but not in Firefox. Firefox will fall back to the static message 'Downloading offline resources..'. Internet Explorer doesn't support the offline application cache as a whole.

I'm sure more creative souls have it in them to build a really elegant and visually pleasing progress indication using this technique. I'm curious to hear (or see!) how you would represent the update process.

Check for local file browsing with JavaScript

2012-04-02T15:34:00.000+02:00

Because I do most of my research while commuting by train, I often pull entire websites offline using httrack. While browsing the jQuery Mobile documentation locally this morning, I stumbled upon following gem.

I was curious to see how they determine whether a page is browsed locally or not. Looking into the source, I was a bit dissapointed to find nothing but plain common sense. The trick is comparing the protocol of the current location with known local protocols.

if ( location.protocol.substr(0,4)  === 'file' ||
     location.protocol.substr(0,11) === '*-extension' ||
     location.protocol.substr(0,6)  === 'widget' ) {
    // Disable AJAX support etc
}

If you would want to use that check in multiple locations in your codebase, you might want to extend the location object with an isLocal function.

window.location.constructor.prototype.isLocal = function() { 
    return this.protocol.substr(0,4)  === 'file' || 
            this.protocol.substr(0,11) === '*-extension' || 
            this.protocol.substr(0,6)  === 'widget'; 
}

The function could be used like this.

if (window.location.isLocal()) {
    // Disable AJAX support etc
}

Add images to a GitHub readme

2012-04-01T19:32:00.000+02:00

Today I wanted to add some screenshots to a GitHub readme for the sake of documenting. While this wasn't particularly hard, I do had to iterate a few times before I got it right.

Hosting the images

You could simply add the images to your repository and reference them using the raw url's, but this isn't very efficient. Using this method, every request needs to go through GitHub's application layer. It's far better to make use of GitHub Pages, a feature purely designed to publish web content. I also like how you're not polluting the repository this way.

Referencing the images

I'm using the - oh so beautiful and simple - markdown format for most of my readme's. The syntax for embedding images looks like this.

![My image](username.github.com/repository/img/image.jpg)

I hope this post fills in the void I stumbled upon when googling for useful pointers earlier today.

How a web application can download and store over 2GB without you even knowing it

2012-03-25T13:11:00.000+02:00

I have been experimenting with the HTML5 offline application cache some more over the last few days, doing boundary tests in an attempt to learn more about browser behaviour in edge cases.

One of these experiments was testing the cache quota.

Two weeks ago, I blogged about generating and serving an offline application manifest using ASP.NET MVC. I reused that code to add hundreds of 7MB PDF files to the cache.

public ActionResult Manifest()
{     
    var cacheResources = new List<string>();
    var n = 300; // Play with this number

    for (var i = 0; i < n; i++)
        cacheResources.Add("Content/" + Url.Content("book.pdf?" + i));

    var manifestResult = new ManifestResult("1")
    {
        NetworkResources = new string[] { "*" },
        CacheResources = cacheResources
    };

    return manifestResult;
}

I initially tried adding 1000 PDF files to the cache, but this threw an error: Chrome failed to commit the new cache to the storage, because the quota would be exceeded.

After lowering the number of files several times, I hit the sweet spot. I could add 300 PDF files to the cache without breaking it.

Looking into chrome://appcache-internals/, I can see the size of the cache being a whopping 2.2GB now for one single web application.

As a user, I had no idea that the website I'm browsing is downloading a suspicious amount of data in the background. Chrome (17.0.963.83), nor any other desktop browser that I know of, warns me. I would expect the browser to ask for my permission when a website wants to download and store such an excessive amount of data on my machine.

Something else I noticed, is that other sites now fail to commit anything to the application cache due to the browser-wide quota being exceeded. I'm pretty sure this 'first browsed, first reserved' approach will be a source of frustration in the future.
To handle this scenario we could use the applicationCache API to listen for quota errors, and inform the user to browse to chrome://appcache-internals/ and remove other caches in favor of the new one. This feels sketchy though; shouldn't the browser intervene in a more elegant way here?

What are your thoughts? What would you want your browser to do in these scenarios?

Sent from my phone

2012-03-17T19:46:00.000+01:00

According to Matt from 37signals the line "Sent from my iPhone" at the bottom of an email means this:

Let’s be honest. “Sent from my iPhone” really means “I’m not going to bother to proofread and correct this because it would take me an extra 30 seconds.”

I agree. I too use this line as an excuse to write a terse message and omit proper salutations.

However, I also think these four simple words greatly helped the viral growth of the mobile phone. Having early adopters brag about how they're sending emails from their fancy new phone must have been an invaluable form of word-of-mouth advertising.

I think it's safe to say that those four words first are a marketing technique and then a convenient way of letting your correspondents know that there might be a typo in your message. If not, the default line wouldn't include the brand name.

I'm turning this thing into a rambling though. What I really wanted to share is a snippet of the much recommended book The Lean Startup, where Eric Ries shares a story on how Hotmail used a similar technique in 1996 to fuel their viral engine of growth.

But everything changed when they made one small tweak to the product. They added to the bottom of every e-mail the message "P.S. Get your free e-mail at Hotmail" along with a clickable link.
Within weeks, that small product change produced massive results. Within six months, Bhatia and Smith had signed up more than 1 million new customers. Five weeks later, they hit the 2 million mark. Eighteen months after launching the service, with 12 million subscribers, they sold the company to Microsoft for $400 million.

I find it incredibly intriguing how a few words can have such an impact. Maybe you do too.

HTML5 Offline Web applications as an afterthought in ASP.NET MVC

2012-03-14T15:39:00.000+01:00

Recently I prototyped a mobile web application using ASP.NET MVC, jQuery Mobile and some HTML5 features. One of the key goals was to find out how far you can push a web 'application' until the browser starts getting in the way. Working disconnected is one of these things that appear to be a major showstopper at first.

However - to my surprise honestly - the HTML5 Offline Web applications API seems to be widely implemented across modern browsers already. Not of all of them though. Looking into the specifics, the API itself is fairly straightforward. At his core, you will find the manifest file, which dictates which files should be cached by the browser. The API provides other useful events and methods for inspecting the status of the cache and swapping the cache for a newer version, but they are out of scope today. A useful resource to read up on the full API can be found here, and a working example implementation can be found here.

The manifest file

Back to the manifest file. A manifest file could look like this.

The first line in the file should say CACHE MANIFEST. If you want to write comments, you should prefix the lines with a number sign.

In the CACHE section you declare which files should be cached. An important and interesting note is that these files will be served from the cache, even if you're online.

In the NETWORK section you declare which files the browser should try to download from the server, regardless of whether the user is online or offline.

In the last section, the FALLBACK section, you can define fallback resources to be used when the user is offline.

Serving and generating the manifest file

Now that we got all this theory out of the way, let's look at generating and serving the manifest file using ASP.NET MVC.

I started by adding a ResourcesController with one action named Manifest.

public class ResourcesController : Controller
{             
    public ActionResult Manifest() { }
}

This action should serve a text file, using a specific cache-manifest MIME type. To accommodate this I created a new action result, which inherits from the FileResult class, and overwrites the content type.

public class ManifestResult : FileResult
{
    public ManifestResult(string version)
        : base("text/cache-manifest") { }    
}

I also made this same class (for the sake of example) responsible for formatting and writing the manifest file to the output stream. That's why I added a few extra properties to the manifest result, one for each section and one for versioning. Versioning the file comes in handy when you want to expire the cache, because it only expires when the manifest file changes.

public class ManifestResult : FileResult
{
    public ManifestResult(string version)
        : base("text/cache-manifest")
    {
        CacheResources = new List<string>();
        NetworkResources = new List<string>();
        FallbackResources = new Dictionary<string, string>();
        Version = version;
    }

    public string Version { get; set; }

    public IEnumerable<string> CacheResources { get; set; }

    public IEnumerable<string> NetworkResources { get; set; }       

    public Dictionary<string, string> FallbackResources { get; set; }        
}

To write the file to the output stream, I had to override the WriteFile method.

protected override void WriteFile(HttpResponseBase response)
{
    WriteManifestHeader(response);            
    WriteCacheResources(response);
    WriteNetwork(response);
    WriteFallback(response);
}

private void WriteManifestHeader(HttpResponseBase response)
{
    response.Output.WriteLine("CACHE MANIFEST");
    response.Output.WriteLine("#V" + Version ?? string.Empty);            
}

private void WriteCacheResources(HttpResponseBase response)
{
    response.Output.WriteLine("CACHE:");           
    foreach (var cacheResource in CacheResources)
        response.Output.WriteLine(cacheResource);
}

private void WriteNetwork(HttpResponseBase response)
{
    response.Output.WriteLine();
    response.Output.WriteLine("NETWORK:");            
    foreach (var networkResource in NetworkResources)
        response.Output.WriteLine(networkResource);
}

private void WriteFallback(HttpResponseBase response)
{
    response.Output.WriteLine();
    response.Output.WriteLine("FALLBACK:");
    foreach (var fallbackResource in FallbackResources)
        response.Output.WriteLine(fallbackResource.Key + " " + fallbackResource.Value);
}

In the CACHE section I wanted to include all my static resources, meaning the contents of the Scripts and Content folder. To do this in a simple and low-maintenace fashion I introduced the GetRelativePathsToRoot method. This method takes the path of a virtual folder, recursively scans its content and returns a list of relative paths for each file.

private IEnumerable<string> GetRelativePathsToRoot(string virtualPath)
{
    var physicalPath = Server.MapPath(virtualPath);
    var absolutePaths = Directory.GetFiles(physicalPath, "*.*",   SearchOption.AllDirectories);

    return absolutePaths.Select(
        x => Url.Content(virtualPath + x.Replace(physicalPath, ""))
    );
}

For the Content folder, the result could look something like this.

To add pages to the CACHE section, I used the Url.Action method.

For the NETWORK resources, I added an asterisk, which basically means that the cache shouldn't be used when the user is online. I didn't specify any fallback resources in this example.

public ActionResult Manifest()
{
    var pages = new List<string>();
    pages.Add(Url.Action("SomeAction", "ControllerName"));    

    var scriptsPaths = GetRelativePathsToRoot("~/Scripts/");
    var contentPaths = GetRelativePathsToRoot("~/Content/");

    var cacheResources = new List<string>();
    cacheResources.AddRange(pages);
    cacheResources.AddRange(contentPaths);
    cacheResources.AddRange(scriptsPaths);
    
    var manifestResult = new ManifestResult("1.0")
    {
        NetworkResources = new string[] { "*" },
        CacheResources = cacheResources
    };            

    return manifestResult;
}

Setting up a route and including the manifest

Now that we are able to generate and serve a manifest file, we should set up a specific route for the manifest file; some browsers aren't very forgiving and expect it to have a specific name and location: /cache.manifest.

routes.MapRoute("cache.manifest", "cache.manifest", new { controller = "Resources", action = "Manifest" });

The last step I had to take was include a reference to the manifest file in the html element.

<html manifest="@Url.RouteUrl("cache.manifest")")/>

Poor man's testing

To verify if all of this works, you can look at the console of the Chrome developer tools. You should see something like this.

That console logging has proven to be extremely useful when debugging the manifest file.

You could also just browse to the manifest file to inspect its content. Don't mind this screenshot too much, obviously there's plenty of cleaning up to do in my Scripts folder.

Summary

In this post I showed you a technique I came up with to take advantage of ASP.NET MVC to easily generate, maintain and serve an HTML5 Offline Webappliction manifest file:

Create a controller and action that can serve the file
Create a new action result, which returns the correct MIME type and formats the file
Set up a specific route
Include a reference to the manifest in the html tag

Remember, this is a proof of concept, it's not perfect. I look forward to any feedback you might have!

Learning: the Hacker Way

2012-03-11T11:08:00.000+01:00

I have had a fair amount of discussions on continuous learning and knowledge sharing the past few days. It became rather obvious that a lot of us have developed their own techniques, but also that maybe most of us are still in search of more efficient techniques. Having gone through several phases myself, I would like to share my current way of learning: the Hacker Way.

Here are some snippets taken from a recent letter from Mark Zuckerberg addressed to the Facebook shareholders.

Hacking is an inherently hands-on and active discipline. Instead of debating for days whether a new idea is possible or what the best way to build something is, hackers would rather just prototype something and see what works. There's a hacker mantra that you'll hear a lot around Facebook offices: Code wins arguments.

I like to believe learning should be a hands-on activity as well. Basically, stop consuming, start producing. Don't get me wrong, I do think there is value in reading blog posts (I might be slightly biased on this one), reading books and watching videos, but I find that this value is marginal compared to what you gain by actually doing it.

I remember I wanted to step up my JavaScript game two years ago, and ordered a book. A few months after finishing the book, I finally had the chance to implement something in JavaScript, but I couldn't. I remembered some syntax, and most concepts, but I couldn't do it. Eventually I really learned JavaScript by doing it on the job, having to constantly fall back on the book and Stackoverflow, making costly mistakes along the way.

It's not a terrible idea to pick up a new technology on the job, but it very much depends on the environment. If you're in the consultancy business, there often is no room to pick up a brand new technology on the job. Customers expect you to know what you're doing.
Imagine you really want to get into mobile development, and you were able to get in the race for a new mobile project. Getting interviewed by the customer, you have to admit that you haven't built something for mobile yet, but you did read a book and you find mobile development really interesting. It's really hard to sell that. If you are able to show something you made, and talk about things that work and things that don't work in the real world, the customer can get a far better feel if you would be a good fit for the project.

The Hacker Way is an approach to building that involves continuous improvement and iteration. Hackers believe that something can always be better, and that nothing is ever complete.

Hacker culture is also extremely open and meritocratic. Hackers believe that the best idea and implementation should always win - not the person who is best at lobbying for an idea or the person who manages the most people.

This is where it becomes interesting and where you can really make the difference as a company.

Depending on various parameters, you often don't have the room to experiment on the day job, but if you're building something by yourself or in a small team without these constraints, you can get wild: experiment, innovate and fail often. Not only do you build experience faster, but you also (maybe indirectly) challenge existing practices and build a deeper understanding of the used methodologies and technologies. Also, having the freedom to innovate, aspiring to get closer to the Silver Bullet, can bring yourself, the company and the industry to the next level.

This is just the first step of the cycle though. The next step should be to get these little projects and acquired experiences out there. Share them with your peers and the community, educate those who want to listen and ask for feedback from those who care, hopefully fueling the inspiration of others.

A welcome side-effect might be that the results could be used to extend your personal and your company's portfolio. This can help you prove that you can do more than just talk the walk, talk is cheap after all.

I want to know from you what the flaws are in my way of thinking. Is the described technique naive, too idealistic or unrealistic?
Which technique has proven to work best for you?

ASP.NET MVC4 bundling in ASP.NET MVC3

2012-02-25T20:26:00.000+01:00

One of the new wildly evangelized features of ASP.NET MVC4 is the built-in support for bundling and minification of scripts and stylesheets.

I don't see any reason why this new feature wouldn't work for ASP.NET MVC3 though. If you open the packages config of an ASP.NET MVC4 beta project, you will find that bundling support lives in the Microsoft.Web.Optimization package.

<package id="Microsoft.Web.Optimization" version="1.0.0-beta" />

So we should just be able to install this package for an ASP.NET MVC3 project. To install the package, run following command. Pay attention to the -Pre switch.

PM> Install-Package Microsoft.Web.Optimization -Pre
Attempting to resolve dependency 'Microsoft.Web.Infrastructure (= 1.0.0)'.
Successfully installed 'Microsoft.Web.Infrastructure 1.0.0.0'.
Successfully installed 'Microsoft.Web.Optimization 1.0.0-beta'.
Successfully added 'Microsoft.Web.Infrastructure 1.0.0.0' to Optimization.
Successfully added 'Microsoft.Web.Optimization 1.0.0-beta' to Optimization.

Adding bundles happens when the application starts, together with registering areas, adding global filters and registering routes.

The quickest way to enable bundling is by enabling the default bundles.

BundleTable.Bundles.EnableDefaultBundles();

This method will add two bundles to the bundle table: one bundle for the stylesheets in the Content folder and one bundle for the scripts in the Scripts folder. The default bundles try to take core scripts into account when ordering the scripts in the bundle. For example, jQuery will be included before any of its plug-ins are included.

To reference these bundles you can add following snippet to your view or layout file.

<link href="@System.Web.Optimization.BundleTable.Bundles.ResolveBundleUrl("~/Content/css")" 
    rel="stylesheet" type="text/css" />
<script src="@System.Web.Optimization.BundleTable.Bundles.ResolveBundleUrl("~/Scripts/js")">
</script>

If you start your application now, and inspect the HTML, you will find two versioned links to a minified version of your CSS and JavaScript.

If you're trying this on a new project, you will probably have no problems. However, if you're trying this on an existing project, chances are that some things are not included how they should be.

To troubleshoot what's going wrong, you can inspect the results of the GetRegisteredBundles method.

var registeredBundles = BundleTable.Bundles.GetRegisteredBundles();

If you need more fine-grained control, you can remove the default bundles again and add your own bundles to the bundle table.

For example, this is how you can add a jQuery bundle.

var jQueryBundle = new Bundle("~/Scripts/jquery", new JsMinify());
jQueryBundle.AddDirectory("~/Scripts", "jquery*.js", searchSubdirectories: false, throwIfNotExist: true);

BundleTable.Bundles.Add(jQueryBundle);

<script src="@System.Web.Optimization.BundleTable.Bundles.ResolveBundleUrl("~/Scripts/jQuery")">
</script>

When you instantiate a new bundle, specify the relative path of the bundle and pick a bundle transformation. You can add a directory to the bundle, filtered by a search pattern. You can also tell the algorithm to search in the subdirectories or to throw an exception when the directory doesn't exist.

If you don't want to add a whole directory to the bundle, but just one or more files, you can use the AddFile method.

For example, this is a separate bundle for modernizr.

var modernizrBundle = new Bundle("~/Scripts/modernizr", new JsMinify());
modernizrBundle.AddFile("~/Scripts/modernizr-1.7.js", throwIfNotExist: true);

BundleTable.Bundles.Add(modernizrBundle);

Conclusion

It's relatively easy to take advantage of bundling in ASP.NET MVC3. Install the NuGet package, set up the bundle table, include the references in your view or layout page and you're done.

There are some more interesting things you can do using bundling. I just started experimenting with it, so I wouldn't be surprised if I will be writing a few more things on bundling in the coming weeks.

There's no place for monogamy in technology

2012-02-20T08:43:00.000+01:00

In this post I would like to share some of my thoughts on a recent post by James Hague titled 'Don't Fall in Love With Your Technology'. If you haven't read that post yet, please do, it's so short that me summarizing it here would be silly.

I think there is nothing wrong with falling in love with your technology per se. If you want to have a fair relationship with your technology, you have to invest in her. Don't have a superficial relationship, take her home with you, spend some cosy Sunday afternoons together. Get to know her inside out. Know when it's fun to be with her, but also more importantly when it's not.

The problem with falling in love with your technology is that a lot of us fall blindly in love. Please don't. Be unfaithful, do - at least - look at other technologies. Take them out on a crazy Friday night date. In the meanwhile you might discover that that new technology does a few things better than the current one and that she's a lot more fun.

I guarantee that what now seems like the love of your life, will one day be nothing more than a long forgotten memory which might bring a smirk on your face on remembering her, but there will be no regret in leaving her.

ASP.NET MVC4 Installer is incompatible with Microsoft .NET Framework 4.5

2012-02-19T16:52:00.000+01:00

I tried installing ASP.NET MVC4 beta today, but seconds into the installation the WebPI already halted the process.

ASP.NET MVC4 Installer is incompatible with Microsoft .NET Framework 4.5

Apparently, this was documented in the ASP.NET MVC4 installation notes.

This release is not compatible with the .NET Framework 4.5 Developer Preview. You must uninstall the .NET 4.5 Developer Preview before installing the ASP.NET MVC 4 Beta.

I guess you have to be forgiving if you want to play with the early bits.

Make sure to uninstall all the .NET Framework 4.5 related stuff though. I had some pain, wasting over an hour, after only partially removing the installation.

Update: Brad Wilson answers the question why they aren't compatible.
Update: Scott Guthrie announced that the next Beta release should be compatible.

Book review: Working with NHibernate 3.0

2012-02-08T20:21:00.000+01:00

It's been a while since I wrote my last book review, mostly because I'm still trying to figure out when it adds value to write one. For this one it was pretty obvious, there are far too little reviews out there.

Being new to NHibernate, and NHibernate being known as having a steep learning curve, I thought it would be a good idea to do some reading. Searching for books on NHibernate 3.0 on Amazon only yielded three results: NHibernate 3 beginner's guide, NHibernate 3.0 cookbook and working with NHibernate 3.0. None of these books have a decent amount of reviews, so I had to pick judging by the cover and summary. I chose the last one.

The book Working with NHibernate 3.0 by Benjamin Delcamp Perkins contains six chapters, covered over 213 pages.

The first chapter very briefly explains what an ORM is, and then starts looking at configuring NHibernate. This means setting up the session factory and its configuration, creating entities and mapping them to the database (using XML and code), but also configuring log4net (NHibernate's logging framework), serializing startup and interceptors and events. These last two subjects felt a little misplaced in this chapter though.

In the first chapter the foundation for the Guitar store example application is laid. This example application is written in WPF, which bothered me a bit. Not because I dislike WPF, but because when I'm reading a book on a data access technology I really don't want it to be littered with fragments of XAML and code-behind. To make matters worse, the author suggests using a console application to test your queries. I think it would have been far more valuable to use real unit tests to prove the queries are correct, more like this. Another complaint about the example application that I read in another review is that there are no scripts available to set up the database, which might be discouraging if you want to follow along.

Chapter two, three and four cover the various NHibernate query API's: HQL (Hibernate Query Language), ICriteria and LINQ. Every chapter implements the same or at least similar queries. Concepts covered in these chapters are simple queries, complex queries, detached queries, futures and aggregates. I think these chapters succeed in giving a good overview of the ways you can use NHibernate to query data. Also the tips on how to use futures, various fetch modes, the stateless session and aggregates to improve performance will prove useful in the future.

The fifth chapter covers managing state and saving data. In this chapter, the author explains the various ways you can handle database concurrency, listing the advantages and disadvantages of each option. NHibernate caching is also explained, looking at first and second-level caching. Further in this chapter, you can find an example of a custom data type implementation. Finally we arrive at saving data with NHibernate. Next to the standard way of saving data, the author explains the use of Evict, Merge and Persist.

The last chapter, covering only 9 pages, shows how you should set up NHibernate in an MVC3 application.

Conclusion

Although this book doesn't do a great job showing you how to use NHibernate in the real world, it does do a decent job giving you a basic overview of NHibernate's capabilities. Reading this book when you're new to NHibernate will save you from a few costly common NHibernate pitfalls. I don't think I will be able to use this book for reference, but it should be easier now - knowing the correct terminology - to search in the NHibernate documentation online.

My rating: 3/5. Do you advise other books on NHibernate?

Testing DI bootstrappers

2012-02-06T08:59:00.000+01:00

While your Dependency Injection bootstrappers - being responsible for gluing your application together - are a vital part of your application, they are seldom put under test. I don't see any reason why they shouldn't be though. The cost of these tests is negligible, definitely if you compare it to the cost of the often catastrophical outcome of bugs in your bootstrappers.

I encourage you to take a look at the commit history of your DI bootstrappers; I bet they change a lot. Wouldn't it be nice to have a set of tests that proves that the dependency container still behaves like you expect it to at runtime? Next to proving correctness, I think writing these tests also helps you discover various behaviours of your DI container, which is a valuable investment in itself.

Let me show you a few tests that I wrote to put my ASP.NET MVC Ninject bootstrapper under test.

I started by opening up the Ninject bootstrapper, making the CreateKernel method public.

public static IKernel CreateKernel()
{
    var kernel = new StandardKernel();
    
    kernel.Bind<IEntryService>().To<EntryService>();

    return kernel;
}

In the test class, I used the TestInitialize attribute to initialize a new instance of the kernel before every test. I'm not sure this is really necessary, but I want to avoid that my tests experience side-effects of a previous test.

[TestInitialize]
public void Setup()
{         
    _kernel = NinjectMVC3.CreateKernel();
}

Needless to say, the first test should prove that an implementation of the IEntryService interface can be resolved. The behaviour I observed while playing with this case, is that Ninject throws a Ninject.ActivationException when the implementation can't be resolved.

Ninject.ActivationException: Error activating IEntryService
No matching bindings are available, and the type is not self-bindable.
Activation path:
Request for IEntryService

Suggestions:

Ensure that you have defined a binding for IEntryService.

If the binding was defined in a module, ensure that the module has been loaded into the kernel.

Ensure you have not accidentally created more than one kernel.

If you are using constructor arguments, ensure that the parameter name matches the constructors parameter name.

If you are using automatic module loading, ensure the search path and filters are correct.

So to test whether an implementation can be resolved, I just make sure no exceptions are thrown on resolving the dependency.

[TestMethod]
public void Test_IEntryService_Can_Be_Resolved()
{
    AssertDoesNotThrowWhenResolved<IEntryService>();
}

The AssertDoesNotThrowWhenResolved method is a helper method which tries to resolve a dependency of T and asserts that no exceptions are thrown while doing so. The assertion is borrowed from Xunit (package available on Nuget).

private void AssertDoesNotThrowWhenResolved<T>() 
{
    Xunit.Assert.DoesNotThrow(() => _kernel.Get<T>());
}

A second useful test is testing the lifetime of the resolved implementation. For most of my dependencies, I expect a new instance every time they are resolved. This test looks like this.

[TestMethod]
public void Test_IEntryService_Is_New_Instance()
{
    AssertNewInstanceIsResolved<IEntryService>();
}

The AssertNewInstanceIsResolved method is another helper method which resolves two instances of T and asserts they are not the same.

private void AssertNewInstanceIsResolved<T>()
{
    var instance = _kernel.Get<T>();
    var secondInstance = _kernel.Get<T>();

    Assert.AreNotSame(instance, secondInstance);
}

That's it. While these tests are very cheap to write, they do provide great value. I can imagine testing more complex bindings, like contextual bindings, taking a bit more time to set up, but the value of these tests increases proportionally.

Do you put your DI bootstrappers under test? If you don't, why not?

Adding ELMAH to an AppHarbor application

2012-02-04T15:17:00.002+01:00

For those who haven't heard of ELMAH yet, here is the project description.

ELMAH (Error Logging Modules and Handlers) is an application-wide error logging facility that is completely pluggable. It can be dynamically added to a running ASP.NET web application, or even all ASP.NET web applications on a machine, without any need for re-compilation or re-deployment.

While ELMAH is completely not AppHarbor specific, there do seem to be a fair amount of questions on the AppHarbor support forums on how to get ELMAH running. I just installed and configured ELMAH for one of my AppHarbor ASP.NET MVC3 applications, so I thought it would be nice to share and give something back to the AppHarbor support guys.

I think enabling ELMAH shouldn't take more than 10 minutes if you follow this tutorial.

Installation

The easiest way to add ELMAH to your web application is by installing the nuget package.

PM> Install-Package elmah
Attempting to resolve dependency 'elmah.corelibrary (= 1.2)'.
Successfully installed 'elmah 1.2.0.1'.
Successfully added 'elmah 1.2.0.1' to Docary.

Once this is done you will find a newly added Elmah reference and a changed web.config.

I'm not going to list all the changes to the web.config here. If you're interested in what the package does to your web.config, you should just run a git diff.

Error log storage

You have to tell ELMAH where it should store the logs. A list of error log implementations can be found on the project wiki.

I choose to use the in-memory error log for now. Add the elmah section and errorLog element to your web.config.

<elmah>    
    <errorLog type="Elmah.MemoryErrorLog, Elmah" size="250" />
</elmah>

Remote access

By default you can only see the logging on the local machine. To enable remote access, you have to add the security element to the elmah section.

<elmah>
    <security allowRemoteAccess="1" />
    <errorLog type="Elmah.MemoryErrorLog, Elmah" size="250" />
</elmah>

Security

Now that you're allowing remote access, you want to secure the logging page. I used ASP.NET authorization to achieve this.

Add the following element to your web.config, and make sure you change the authorization configuration to the relevant values.

<location path="elmah.axd">
    <system.web>
      <httpHandlers>
        <add verb="POST,GET,HEAD" path="elmah.axd" type="Elmah.ErrorLogPageFactory, Elmah" />
      </httpHandlers>
      <authorization>
        <allow roles="SuperUser"/>
        <deny users="*" />
      </authorization>
    </system.web>
    <system.webServer>
      <handlers>
        <add name="ELMAH" verb="POST,GET,HEAD" path="elmah.axd" type="Elmah.ErrorLogPageFactory, Elmah" preCondition="integratedMode" />
      </handlers>
    </system.webServer>
</location>

Finished

Push these changes, and you should be done. Navigate to elmah.axd in the root of your web application and you should see something like this.

It should be easier now to start playing with more advanced ELMAH options.

A solar storm anecdote

2012-02-01T08:43:00.001+01:00

Last week, several news channels reported on the strongest solar storm since 2005. This news item reminded me of a peculiar support ticket we received one gray Monday morning a few years ago, when I was still writing software for fire departments.

*********************************************************************************
Ticket 7238
Subject: AVL broken
Status: New
Description
06:22 Vehicles stay mostly stationary on the map, even when we are
positive they are en route.
*********************************************************************************

Fire departments that have to cover a large area - and are wealthy enough - often use AVL (Automatic Vehicle Location) to track their vehicles and visualize them on a map. This is extremely valuable, because you always want to dispatch the vehicles with the smallest response time to a high priority intervention. Also being able to advise drivers of possible blockages and toxic gas clouds can save lives. To be able to track a vehicle, an AVL module is installed into each vehicle's cockpit. This module uses GPS to determine the location and sends the location data over GPRS to a central server.

On our end, we had a third party service listening for those location packets and translating them into a more understandable format. This service, not being mission critical, wasn't being monitored, so we had to look into the logs to see what was going on. Scrolling through megabytes of debug logs, we couldn't find anything suspicious.

While I was investigating this, a co-worker had come in and was reading through his mails while sipping on his morning coffee. After reading support ticket 7238, he nonchalantly said he knew what was going on with AVL and he would take over from there.

*********************************************************************************
Ticket 7238
Subject: AVL broken
Status: Pending
Description
08:45 I heard on the radio that there is a solar storm going on at
the moment, which affects sattelites. The AVL module might have
a hard time getting a GPS fix. This issue should solve itself over
time. We will keep an eye on this issue.
---------------------------------------------------------------------------------
06:22 Vehicles stay mostly stationary on the map, even when we are
positive they are en route.
*********************************************************************************

A few days passed and although the solar storm was over, we were still seeing signifcant packet loss. After spending a few hours working the ticket, in which we restarted the service, monitored network traffic on the machine and conctacted the telephony provider, we were getting a bit desperate. We were discussing other potential paths to investigate, when one of our more seasoned co-workers asked "You guys did try restarting the server, right?".

Good enough, after restarting the server, we were seeing no more packet loss and the vehicles started moving on the map again.

*********************************************************************************
Ticket 7238
Subject: AVL broken
Status: Resolved
Description
15:47 The solar storm must be over. The vehicle locations are
being updated in a timely fashion again.
--------------------------------------------------------------------------------
08:45 I heard on the radio that there is a solar storm going on at
the moment, which affects sattelites. The AVL module might have
a hard time getting a GPS fix. This issue should solve itself over
time. We will keep an eye on this issue.
--------------------------------------------------------------------------------
06:22 Vehicles stay mostly stationary on the map, even when we are
positive they are en route.
*********************************************************************************

Until today, we never talked about ticket 7238 again.

How Wikipedia uses HTML5 to save bandwidth

2012-01-19T20:56:00.000+01:00

Something I hadn't noticed until recently is that Wikipedia tries to use the browser's native SVG support to render certain images. For example, if you search for a high resolution image of your country's flag, you will probably end up viewing an SVG. Wikipedia also offers downloads to the image rendered as a PNG though.

Next to being able to scale to an arbitrary size without suffering data loss, the SVG data format allows images to be far more compact. Basically, SVG is just XML, which also means it can be easily compressed to make its size even smaller. For example, this is the (uncompressed) SVG for the flag of the Kingdom of Belgium.

<svg xmlns="http://www.w3.org/2000/svg" width="450" height="300">
    <rect width="450" height="300"/>
    <rect x="150" width="150" height="300" fill="#FAE042"/>
    <rect x="300" width="150" height="300" fill="#ED2939"/>
</svg>

This svg node only weighs as much as 224 bytes, while the image rendered as a high resolution PNG weighs 13.402 bytes. Stuff like that makes a significant difference when you're serving millions of page views on a daily basis.

The first time I touched SVG was a few years ago when I was still working on fire department projects. We were working with a third party that used SVG to draw maps in the browser. Being already in the post-Google Maps era, I thought it was terrible I had to download Adobe's SVG viewer. While the Google Maps technology already works great, there are still things SVG can do better and cleaner, especially for more specialized GIS applications. There is an interesting paper on that subject here, it's a bit outdated though.

I can only applaud making the browser more capable, and losing yet another plug-in. I'm curious to see how other applications will start taking advantage of the opportunities native SVG support across all modern browsers (even Internet Explorer) presents.

Have you already built things using SVG? Or even considered it?

Undoing the Wikipedia black-out

2012-01-18T10:29:00.001+01:00

Update: A comprehensive collection of ways to undo the black-out can be found in this Gist.

Five minutes into work this morning, I already ran against the Wikipedia black-out. If you try it yourself, you will see the article's content being briefly visible and then disappearing.

Curious as I am, I looked at what they were doing in the browser. Upon inspecting the source, you instantly notice that the content is still present in the DOM, and that the black-out is nothing more than an overlay.

If you run following script in the console, you can undo the black-out. This script removes the overlay from the DOM, and makes all other elements visible again (which might not be completely correct).

jQuery("#mw-sopaOverlay").remove(); 
jQuery("*").show();

This was mostly just a fun exercise, I will not be using this script. I fully support this protest.

Autocorrecting unknown actions using the Levenshtein distance

2012-01-15T16:04:00.000+01:00

This weekend I prototyped an idea I had earlier this week: autocorrecting unknown actions in ASP.NET MVC.

Handling unknown actions

To give you an example, let's say I have a Home controller with an action named Kitten on it. If there is an incoming route for the Home controller with Kitty (instead of Kitten) as the action name, the controller will not be able to invoke any action method and instead will call the HandleUnknownAction method.

Here is the snippet from the ASP.NET MVC source.

protected override void ExecuteCore() {
    PossiblyLoadTempData();
    try {
        string actionName = RouteData.GetRequiredString("action");
        if (!ActionInvoker.InvokeAction(ControllerContext, actionName)) {
            HandleUnknownAction(actionName);
        }
    }
    finally {
        PossiblySaveTempData();
    }
}

The HandleUnknownAction is virtual, meaning we can override it in our derived controller. The base implementation of the HandleUnknownAction method does nothing more than throwing a 404 HttpException.

protected virtual void HandleUnknownAction(string actionName) {
    throw new HttpException(404, String.Format(CultureInfo.CurrentCulture,
        MvcResources.Controller_UnknownAction, actionName, GetType().FullName));
}

So let's override the HandleUnknownAction method and try to autocorrect the unknown action name. To be safe, we will only attempt to autocorrect the action name when it's a GET HTTP request.

protected override void HandleUnknownAction(string actionName)
{
    if (!HttpContext.Request.HttpMethod.Equals("GET", StringComparison.OrdinalIgnoreCase))
        Throw404HttpException(actionName);
    
    TryToRedirectToAnActionNearby(actionName);           
}

Listing all actions

First we need a list of all available action names. I reflect on the methods of the current controller and select the methods which are public, can be invoked and are an instance method. Also the method should return an ActionResult, not be decorated with the HttpPost attribute and not have a special name. I'm pretty sure I'm missing a few things here, but there seems to be no generic way to extract this metadata from a controller. Places where these type of things are used in the framework seem to be internal or non-public.

private IEnumerable<string> GetAllHttpGetActionNames()
{
    return GetType()
            .GetMethods(BindingFlags.InvokeMethod | 
                        BindingFlags.Public | 
                        BindingFlags.Instance)
            .Where(m => m.ReturnType == typeof(ActionResult) &&
                        !m.IsSpecialName &&
                        !m.GetCustomAttributes(true)
                            .Contains(typeof(HttpPostAttribute)))
            .Select(m => m.Name)
            .Distinct();
}

Once we have all these action names, we want to see how distant they are from the unknown action name we are trying to autocorrect here. To calculate this we can use the Levenshtein distance algorithm.

The Levenshtein distance

The Levenshtein distance is defined by Wikipedia like this.

In information theory and computer science, the Levenshtein distance is a string metric for measuring the amount of difference between two sequences. The Levenshtein distance between two strings is defined as the minimum number of edits needed to transform one string into the other, with the allowable edit operations being insertion, deletion, or substitution of a single character. It is named after Vladimir Levenshtein, who considered this distance in 1965.

An implementation of this algorithm in C# could look like this.

public static int CalculateDistance(string str1, string str2) 
{
    var matrix = new int[str1.Length + 1, str2.Length + 1];

    for (var i = 0; i <= str1.Length; i++)
        matrix[i, 0] = i;
    for (var j = 0; j <= str2.Length; j++)
        matrix[0, j] = j;

    for (var i = 1; i <= str1.Length; i++)
    {
        for (var j = 1; j <= str2.Length; j++)
        {
            var cost = str1[i - 1] == str2[j - 1] ? 0 : 1;

            matrix[i, j] = (new[]
            {
                matrix[i - 1, j] + 1, matrix[i, j - 1] + 1, matrix[i - 1, j - 1] + cost
            }).Min();

            if ((i > 1) && 
                (j > 1) && 
                (str1[i - 1] == str2[j - 2]) &&
                (str1[i - 2] == str2[j - 1]))
            {
                matrix[i, j] = Math.Min(matrix[i, j], matrix[i - 2, j - 2] + cost);
            }
        }
    }

    return matrix[str1.Length, str2.Length];
}

This is a direct port from the pseudocode found on Wikipedia. These tests might, probably a lot more than the implementation, help you understand what the Levenshtein algorithm calculates.

[TestMethod()]
public void Test_CalculateDistance_With_Two_Empty_String()
{           
    Assert.AreEqual(0, Levenshtein.CalculateDistance(string.Empty, string.Empty));
}   

[TestMethod()]
public void Test_CalculateDistance_With_Empty_First_String()
{         
    Assert.AreEqual(6, Levenshtein.CalculateDistance(string.Empty, "kitten"));
}

[TestMethod()]
public void Test_CalculateDistance_With_Empty_Second_String()
{           
    Assert.AreEqual(6, Levenshtein.CalculateDistance("kitten", string.Empty));
}

[TestMethod()]
public void Test_CalculateDistance_With_Missing_Characters()
{           
    Assert.AreEqual(2, Levenshtein.CalculateDistance("kitten", "kitt"));
}

[TestMethod()]
public void Test_CalculateDistance_With_Wrong_Characters()
{           
    Assert.AreEqual(1, Levenshtein.CalculateDistance("kitten", "kittyn"));
}

[TestMethod()]
public void Test_CalculateDistance_With_Too_Much_Characters()
{           
    Assert.AreEqual(5, Levenshtein.CalculateDistance("kitten", "kittenkitty"));
}

[TestMethod()]
public void Test_CalculateDistance_With_Equal_Strings()
{          
    Assert.AreEqual(0, Levenshtein.CalculateDistance("kitten", "kitten"));
}

Now that we have implemented the Levenshtein distance algorithm, we can calculate the distances between the unknown action name and all the available action names.

private Dictionary<string, int> CalculateLevenshteinDistance(IEnumerable<string> actionList, string actionName)
{
    return actionList
            .Select(a => new
            {
                Action = a.ToLower(),
                Distance = Levenshtein.CalculateDistance(a.ToLower(), actionName.ToLower())
            })                    
            .ToDictionary(k => k.Action, v => v.Distance);
}

For the unknown action name 'Kitty', when the action names 'Kitten', 'Index' and 'Dog' are available, this method would return a dictionary that looks like this.

'kitten' : 2
'index' : 6
'dog' : 6

Putting it all together

Now we have this dictionary, we want to filter on a certain distance threshold. I picked three, given that when a word is three characters off, the chance of it being a typo is rather small.

If the dictionary still contains some items after filtering, we want to take the action with the shortest distance, this action is the nearest to the unknown action. Only thing left to do is change the action in the RouteData and execute a RedirectResult. The easiest way to generate a url to redirect to, is to use the controller's UrlHelper to let it generate the url based on the RouteData.

private void TryToRedirectToAnActionNearby(string actionName)
{
    var httpGetActionNames = GetAllHttpGetActionNames();
    if (!httpGetActionNames.Any())
        Throw404HttpException(actionName);

    var actionDistanceMap = CalculateLevenshteinDistance(httpGetActionNames, actionName)
                                .Where(i => i.Value <= 3);
    if (!actionDistanceMap.Any())
        Throw404HttpException(actionName);

    var shortestDistance = actionDistanceMap.Select(v => v.Value).Min();
    var nearestAction = actionDistanceMap.Where(i => i.Value == shortestDistance).First().Key;

    ControllerContext.RouteData.Values["action"] = nearestAction;

    new RedirectResult(Url.RouteUrl(RouteData.Values), permanent: true)
        .ExecuteResult(ControllerContext);
}

The outcome

Now when I, the user, type http://somesite/someController/kitty, I will be redirected to http://somesite/someController/kitten without me even noticing.

Feedback

This implementation definitely is not production ready. It's a prototype, not even under test. I wonder if this is even something you would want to do. Or is this breaking the Web in one way or the other? Would it bother search engines?

An approach to mobile support in ASP.NET MVC

2012-01-12T21:16:00.000+01:00

I have been spending a decent amount of time working on a side project in ASP.NET MVC lately. From the start, I planned on supporting mobile. There are lots of ways you can use or extend ASP.NET MVC to support mobile. Having tried several, I can say they all have their merits, and the solution that will work for you very much depends on your requirements. In this post I will run you over the implementation that worked for me, for my specific requirements.

I looked at dynamically changing layouts, but I doubt if there is anyone who has gotten a satisfying result with that. I also considered rolling my own view engine, but after going with that for a little while, it became obvious that this wasn't going to work out either. The behavior and functionalities of my mobile version were quickly drifting away of those of the desktop version. I could only reuse so few existing actions or viewmodels that I was close to seeing the two versions as two completely different sites. Let me elaborate a bit further. While it was often possible to use the same viewmodels for both versions, it wasn't as clean as I wanted it to be. I strive for lean views, and doing projections in the views just to get the viewmodels in a workable format felt not right. Also, I had a few cases where I had to split one action in multiple smaller ones to make the flow work for the mobile version.

Eventually I ended up dividing the desktop and mobile version in two separate areas: desktop and mobile. Because some things do overlap, I share a few abstract base controllers between the areas.

So when I look at the root of my application now, I see no more views. Two things are left though: the HomeController and the AccountController.

I was able to pinpoint three points in the application where I need to take special care of detecting the device or rerouting to the correct area.

The default route when the user types http://MySite in the address bar still is Home/Default. In the default action of the HomeController I want to detect the area I want to redirect to, based on the user agent. Notice I'm using a permanent redirect, not wasting a HTTP request.

public class HomeController : Controller
{
    public ActionResult Index()
    {           
        var view = Request.IsAuthenticated ? "Index" : "Welcome";
        var area = Request.ResolveDestinationArea();             

        return RedirectToActionPermanent(view, "Home", new { Area = area });            
    }
}

This is the first scenario. When a user opens the homepage, I redirect him to a specific area based on the user agent. If the user is on a desktop he will be redirected to http://MySite/desktop/home and when he's on a mobile device he will be redirected to http://MySite/mobile/home. If this somewhat well-informed guess turns out to be wrong, the user can still use a link to take him to the correct area.

There are two places left where I have to add some logic assuring the user stays in the correct area. When the session is expired, I have to redirect the user to the correct login page. Since you can only define one loginUrl and there is no way of plugging into that, I let ASP.NET redirect to the Account controller in my root.

<authentication mode="Forms">
  <forms loginUrl="~/Account/LogOn" timeout="2880"/>
</authentication>

In the AccountController I can use the loginUrl in the querystring to decide which area I want to redirect to. If it contains mobile, I redirect to the mobile area. If it contains desktop, I redirect to the desktop area.

public class AccountController : Controller
{       
    public ActionResult LogOn()
    {
        return RedirectToAction(
            "LogOn", "Account", new { Area = Request.ResolveDestinationArea() });      
    }
}

If the url were http://MySite/Account/LogOn?ReturnUrl=%2fmobile%2fentry%2fadd, the 'mobile' substring in the ReturnUrl querystring would suffice to make this work.

Now there is one more place left where we want to take care of redirecting to the correct area: in the error handling. Like always, there are several ways of handling errors in ASP.NET MVC. I chose for the one where I could handle exceptions outside the MVC pipeline and add custom redirect logic easily. I'm talking about the Application_Error event in the Global.asax. When I'm handling this event, I can also use the request url to resolve the correct area.

protected void Application_Error()
{
    Response.TrySkipIisCustomErrors = true;

    var exception = Server.GetLastError();
    var httpException = exception as HttpException;

    Response.Clear();
    Server.ClearError();           
    
    var routeData = new RouteData();
    routeData.DataTokens["area"] = HttpContext.Current.Request.ResolveDestinationArea();
    routeData.Values["controller"] = "Errors";
    routeData.Values["action"] = "General";
    routeData.Values["exception"] = exception;
    
    Response.StatusCode = 500;
    
    if (httpException != null)
    {
        Response.StatusCode = httpException.GetHttpCode();
        if (Response.StatusCode == 403)
            routeData.Values["action"] = "Forbidden";
        if (Response.StatusCode == 404)
            routeData.Values["action"] = "NotFound";
    }

    var errorsController = (IController)new ErrorsController();
    var requestContext = new RequestContext(new HttpContextWrapper(Context), routeData);
    errorsController.Execute(requestContext);
}

The url in this case would look like http://MySite/mobile/home, so the 'mobile' substring in the url should suffice to make the correct decision.

Pay attention, when you assemble a routedata object, you want to store the area value in the DataTokens property.

So far we have identified three things we want to hold in account when resolving the correct area: the url, the querystring of the url and the user agent. These decisions are encapsulated into a ResolveDestinationArea extension method on the HttpRequest class.

public static string ResolveDestinationArea(this HttpRequest request)
{
    var mobileArea = "mobile";
    var desktopArea = "desktop";

    var uri = new Uri(request.Url.AbsoluteUri);
    // We want to search in the left part of the absolute uri
    var valueToSearchIn = uri.GetLeftPart(UriPartial.Path);
    
    // However if there is a returnUrl querystring, we want to search in that value
    var returnUrl = HttpUtility.ParseQueryString(uri.Query).Get("returnUrl");
    if (!string.IsNullOrEmpty(returnUrl))
        valueToSearchIn = returnUrl;
        
    // If the url contains 'mobile', redirect to the mobile area    
    var urlContainsMobile = 
    valueToSearchIn.IndexOf("mobile", StringComparison.OrdinalIgnoreCase) > -1;
    if (urlContainsMobile)
        return mobileArea;

    // If the url contains 'desktop', redirect to the desktop area
    var urlContainsDesktop = 
    valueToSearchIn.IndexOf("/desktop/", StringComparison.OrdinalIgnoreCase) > -1;
    if (urlContainsDesktop)
        return desktopArea;

    // If the url does not contain 'mobile' nor 'desktop', we have to look at the user agent
    return request.Browser.IsMobileDevice ? mobileArea : desktopArea;
}

The implementation of the ResolveDestinationArea method is a bit simplistic. Most sites would want to be a bit more careful about the way the url is parsed. Only the second part of the url really matters (/desktop/ or /mobile/), unless the returnUrl is present in the querystring.

Conclusion

Like I said in the introduction, there are several ways you can use ASP.NET MVC to support mobile devices. I think this pragmatic, maybe naïve solution, works for me. There are only three scenarios where I need to think about mobile devices: when the user enters the homepage, when the user needs to authenticate again and when there is an unhandled exception. All the logic is encapsulated in one little extension method on the request object, making it easily changeable in the future.

There is one extra scenario which I'm thinking of supporting in the future. When the user explicitly chooses a different version, I can save that preference to a cookie, so that I can use that as well to resolve the area in the future.

If you already implemented mobile support in some way, what worked for you?

2011 Annual Review

2011-12-26T21:35:00.000+01:00

Since the end of the year is approaching, I would like to look back on 2011 and take a peek at 2012. Unlike the years before, I'm hardly setting any goals this year. Interests change so quickly, and new opportunities present themselves so regularly, I feel like it might be ignorant to set these things in stone.

My career

The year started out interesting, working on a brand new product for fire departments at Ferranti Computer Systems. Although my role in this project should have been fairly satisfying, I still felt like my position back then failed to fill a certain void. That's why I decided to take the leap into the Great Unknown.

In September, I started working for Euricom as a consultant. So far, I haven't regretted making that decision at all, I ended up in a group of intelligent, progressive and enjoyable people.

Blog

This year I managed to write 69 posts (70 including this one), which is a bit more than the year before.

Plenty of posts have gotten decent attention this year, making the traffic tripple compared to 2010. And no matter how other bloggers claim you shouldn't care about your blog statistics, this does mean a bunch to me.

The main topics I wrote about this year were mostly web related (HTML5, JavaScript and ASP.NET MVC). I even wrote some opinionated posts this year, which I found harder, but also more rewarding.

Last year, I set a goal to get better at MVC and to at least look at WebMatrix. I spent a large part of the year building things in ASP.NET MVC at the day job, so that goal was easily achieved. I built one small thing in WebMatrix, which was enough to give me a feel of what WebMatrix is all about.

My guess is that I will cover similar topics next year. My faith in the web has only grown stronger. I think the rise and rise of the browser and JavaScript provides us with unseen opportunities, but also with an enormous amount of challenges. I am really eager to start looking more into serious JavaScript development.

Community

I think I did fairly well this year. I attended several local community meetings and got to to know a lot of nice people in the process.

The highlight probably was my own talk on WebSockets at HTML5 Webcamps. I really enjoyed that experience. I do not aspire to be a speaker though, but should I have interesting things to talk about, I probably won't reject invitations.

I'm not sure if I will be doing a lot of community events in 2012. I often don't seem to get out of them what I hoped for. Also, Euricom regularly organizes meetups to talk about all things software development. These will already consume plenty of time.

Travelling

This year my girlfriend and I did a three-week roadtrip along the West Coast. This trip was by far the best time of our lives. We are determined to undertake similar projects in the future again.

Next to that, we revisited the Czech Republic, taking plenty of time to visit Prague this time.

We're not sure what 2012 will bring. Right now, we are looking into something more nearby, Italy.

Sport/Health

Last year I picked up boxing, but I had to give that up due to scheduling conflicts. I have been working a tweaked P90X routine twice a week instead, which is very much recommended.

I reasonably succeeded in achieving my running ambitions. Altogether, I ran somewhere around 800 kilometers (= 500 miles) this year. This summer, I probably was in the best shape of my life, running 15 km pretty (= 9 miles) comfortably. Although the graph below suggests that I'm slacking, it's only partially true. I have been doing shorter (and higher paced) runs lately, increasing the intensity. Running is probably one of the sports that give me the most energy in return, so I intend to stay at it.

Conclusion

2011 has been an eventful year for me. In 2012, I hope to be able to keep doing the things I love, surrounded by the people I love. Trying to live life as much as I can.

I wish you and your family the best for 2012. That the new year may bring you health, love and plenty of hacking.