When I started with continuous integration, I used CruiseControl.NET, but the XML configuration quickly got to be really tedious.

It didn’t take me long to switch to Hudson CI.  It’s free, and it’s easily managed through an intuitive web interface. Not that I can’t grok XML configuration files; I’d just rather be spending my time building things instead of figuring out how to get them built.

For a primer on how to use Hudson CI for .NET projects, take a look at Getting Started With CI Using Hudson For Your .NET Projects by Bob Cravens, or Continuous Integration with Hudson and .NET

The only problem is Hudson is Java-based, so it’s not exactly plug-n-play when it comes to .NET, MSBuild, and Visual Studio. Particularly, when a build failed, I would have to scan through hundreds of lines looking for the cause of the error, which is difficult to do with CTRL-F when this line is completely normal:

Compile complete -- 0 errors, 0 warnings

What is needed is the Log Parser Plugin (link includes formatting rules), which allows you to define regular expressions that determine how to interpret the lines of content in the MSBuild output, so that warnings and errors can be called out in different colors the same way that Visual Studio does. This plugin can be installed through Hudson’s built-in plugin manager, but you have to configure it yourself.

First, you need to create a text file on the server that hosts Hudson. If you have a cluster of servers, this must be done on the master server.

Then, define the rules for the Log Parser Plugin to use. For this, it is helpful to have a reference of

MSBuild / Visual Studio aware error messages and message formats handy.

Here are the rules that I came up with, which I saved to C:\hudson\VSParsingRules.txt:

# Divide into sections based on project compile start
start /^------/

# Compiler Error
error /(?i)error [A-Z]+[0-9]+:/

# Compiler Warning
warning /(?i)warning [A-Z]+[0-9]+:/

It’s really not complex.  A line starting with 5 dashes defines the start of a section where MSBuild starts building a project. Then there are rules to define an error and a warning.

Once this file is in place, go to Manage Hudson > Configure System and enter the path to the rules file under Console Output Parsing. Then, in the configuration for each project, under “Post-build Actions”, check “Console output (build log) parsing” and select the appropriate rules from the drop-down.

Once the next build is finished, you will see a new “Parsed Console Output” link in the left-menu on each build.

One last tip: you will want to disable the Automatic Refresh setting (in the top-right corner of your browser window) while viewing parsed log output so that your screen doesn’t flicker and refresh every few seconds, causing you to lose your place.

If you run an ASP.NET website with any non-trivial amount of traffic, the last thing you want to do is try to load an external resource with a web request. Even if you cache it, you’ve taken on that external source as a dependency to your system, which means that your website is only as reliable as the external source you depend on.

The primary strategy I’ve seen to cope with this is to request the data from the remote service on the first request from the web tier, cache it locally, and then have a background process periodically refresh that cache so that subsequent requests can use locally cached data.

Even with this background-updated cache, eventually the external resource will suffer some sort of issue. The data in the cache grows old and is invalidated, and then the web tier starts requesting the data again, except now, 100 different requests try to open 100 different connections to the remote service to get the exact same data, because either they all fail or none of them can complete quickly enough to store the cache item for the other threads to use. The Thread Pool fills up, and ASP.NET hangs its head and gives up.

The Solution

Let’s assume that we are talking about a current weather tease that is a small part of a very large page, and that all our current weather information for the end-user’s zip code comes to us from an external provider.

So how do we avoid this mess? We can do it quite handily with a little NServiceBus and jQuery.

Do not even try to access the data during the standard page load. Instead, ship markup to the client that displays either an empty div or perhaps an AJAX loading spinner. Generate one here, here, here, or a variety of other places if, like me, you just don’t do graphics.

Use jQuery to request the weather data from a web service on your own server, to be returned as JSON. This request should only look for locally cached data sources. If none are found, it should return a JSON response that says "Sorry, I don’t have anything for you now. Please try back later." At the same time, the web page will use NServiceBus to send a DownloadRemoteWeatherDataCmd to a background service.

public ActionResult GetCurrentConditions(string zip)
{
	var conditions = CurrentConditions.GetFromLocalSourcesOnly(zip);

	if (conditions == null)
	{
		// Really we should wrap conditions with a container that we
		// keep in cache, even if the value is null, where we can
		// keep track of how often we send this command, so that we
		// don't flood our back-end service with requests.
		Bus.Send(new DownloadRemoteWeatherDataCmd { ZipCode = zip });
	}

	return Json(new {
		hasData = conditions != null,
		conditions = conditions
	}, JsonRequestBehavior.AllowGet);
}

While your client side has set a timeout to request the data again after a few seconds, the background process has received the DownloadRemoteWeatherDataCmd. Using a Saga, the endpoint manages duplicate requests (from multiple servers in the same server farm, for example) and makes sure that for each unique data request, only one request to the external resource is being made.

public class CurrentWeatherSagaData : ISagaEntity
{
	public string ZipCode { get; set; }
	public DateTime DoNotDownloadAgainUntil { get; set; }

	// Required for NServiceBus Sagas - don't touch!
	public Guid Id { get; set; }
	public string OriginalMessageId { get; set; }
	public string Originator { get; set; }
}

public class RemoteDataAccessSaga : Saga<CurrentWeatherSagaData>,
	IAmStartedByMessages<DownloadRemoteWeatherDataCmd>
{
	public override void ConfigureHowToFindSaga()
	{
		this.ConfigureMapping<DownloadRemoteWeatherDataCmd>(
			data => data.ZipCode,
			cmd => cmd.ZipCode);
	}

	public void Handle(DownloadRemoteWeatherDataCmd cmd)
	{
		this.Data.ZipCode = cmd.ZipCode;

		if (DateTime.UtcNow > this.Data.DoNotDownloadAgainUntil)
		{
			Bus.Send(new RateLimitedDownloadRemoteWeatherDataCmd {
				ZipCode = cmd.ZipCode
			});
			Data.DoNotDownloadAgainUntil = DateTime.UtcNow.AddMinutes(10);
		}
	}
}

When the external request completes, the background endpoint stores the data locally (probably in a database) and publishes an INewWeatherDataAvailable.

public class DataDownloader : IHandleMessages<RateLimitedDownloadRemoteWeatherDataCmd>
{
	public IBus Bus { get; set; }

	public void Handle(RateLimitedDownloadRemoteWeatherDataCmd cmd)
	{
		// Download the data from the weather provider

		// Then, insert it into the local database

		// Then, announce it to everyone who cares
		Bus.Publish<INewWeatherDataAvailable>(e => { e.ZipCode = cmd.ZipCode; });
	}
}

The web tier subscribes to INewWeatherDataAvailable, and drops any cache items it might be holding. The next time the client side hits the web service, it will go back to the database and deliver the newly arrived data.

public class CurrentConditionsCacheManager : IHandleMessages<INewWeatherDataAvailable>
{
	public void Handle(INewWeatherDataAvailable evt)
	{
		string zipCode = evt.ZipCode;

		// Now, drop any cache items related to this zip code
	}
}

In the event of a downtime on the remote resource, compensating actions can be taken on the client side, depending upon business requirements. For example, the spinner could continue to spin indefinitely, or perhaps after 30 seconds, the spinner could be replaced by an apology message that the data couldn’t be loaded right now. As disappointing as that might be, it’s certainly better than an unresponsive web server or Error 503 Server Not Available.

Extra Credit

This is a solid approach, but there are other things we can do to make it even better.

Server/Client Templating

A valid criticism of this approach is that any content delivered via JSON cannot be indexed by search engines. In order to get around this, you can use a templating language that works on the server and the client.  Mustache is just such a language, and is available for JavaScript as well as .NET.

On the server side, if data is available, render the data to HTML so that Google can index it. If it is not, fall back on the AJAX loader image and include the template so that it can be populated the exact same way on the client.

Async Signaling

Once the INewWeatherDataAvailable arrives at the web server, why wait for the client to check back for the update when we can push the data straight to the client? SignalR provides us with the ability to push data directly from the server to the client when it arrives.

I love to write code and I’m a great typist, but I hate typing if it’s not necessary. I’m a big fan of syntactical sugar and I think it’s high time C# got some more of it.

By now we’re all used to seeing this:

public string MyProperty { get; set; }

This eliminates the need to separately declare a private member variable and public property, because the compiler takes care of it for you.

What it doesn’t do is give you the option to set the initial value, like you could with a private member variable. Instead we have to do this:

public class MyClass
{
    public string MyProperty { get; set; }

    public MyClass()
    {
        MyProperty = "initial value";
    }
}

This really becomes more of a pain point now that I have delved into NServiceBus and RavenDB. With NServiceBus, I create a lot of message classes, and when the message includes a collection, it’s usually a good idea to initialize the empty collection so that I don’t get annoying null reference exceptions when I try to use them. The same is true of RavenDB when creating objects with collections to persist to the document store.

What if we could do this?

public string MyString { get; set; default "initial value"; }

public List<int> MyList { get; set; default new List<int>(); }

It would personally save me a LOT of time.

Every year users are conducting more of their lives online, but they’re also doing so from more screens than ever before. It’s not uncommon for me to access the same web application from my work computer, home computer, iPhone, and iPad, and those are just the devices that I consider “mine”. (Not that I own my work computer, but that I am its only user.)

If your web application contains anything of real value, the real danger comes from public computers that your users may use. It’s all too easy for a user to forget to log off from a website, and most webapps make it impossible to log off those sessions remotely.

I believe it’s becoming more important to allow your users the ability to globally log off from your website, clearing not only local credentials stored in cookies, but invalidating all the user’s sessions across all devices.

Some websites have already begun to do this:

• Facebook allows you to remotely log off from other devices, although the functionality is hidden within the Security area under Account Settings, at least it is today. Facebook also shows the approximate location (based on IP) of the activity, as well as a device type. Sometimes this device type is something my mom could understand (Firefox on WinVista, Facebook for iPhone) but in other cases it displays a full browser user agent, which I believe is not helpful for the vast majority of Facebook’s users.  While that is nice for me as a geek, it’s probably overkill for most applications.
• Stack Overflow (and the entire Stack Exchange network of websites) say on their Log Out page that “Clicking Log Out will clear all local credentials in your browser, and log you out on all devices.”

So how to implement global log off?

Solution 1: Session Table

This solution would look a little like the Facebook example.

When the user logs in, write a session row to a database, including the User ID and Session ID. This could include the IP address, location, and user agent information like the Facebook example, but this is not required. Write a session cookie to the user’s browser containing Session ID, salted and encrypted of course.

When the user logs off, delete all session rows for the User ID in order to effect a global log off.

This solution has the advantage that you can support disabling individual sessions without killing off all of them.

Solution 2: User Generation

A simpler solution is to add a Generation concept to your User information. This could be an integer (where every user starts at zero) or a Guid; it really doesn’t matter.

The user cookie should now include both the User ID and the Generation, again salted and encrypted.

Presumably every page request will use a (cached) User object of some type to render the obligatory “Welcome, David” user management header near the top of the page, if nothing else. When fetching the current user, the User object would be loaded by UserID, and then the Generation from the cookie is compared to the Generation in the object. If they match, the login is successful. If not, the cookie is invalid and is discarded.

When the user wants to globally log off, simply increment the Generation on the user object, and then all cookies referencing that User ID will be rendered invalid.

I have a friend.  Let’s call him “Steve”.

Steve was recently complaining to me about the password requirements imposed on him by Corporate IT.  He was using all sorts of words to describe them.  The only one suitable for reprinting was “stupid”.  They went downhill from there rather quickly.

Here are the password requirements Steve has to live with:

• Must contain at least one uppercase letter A-Z.
• Must contain at least one lowercase letter a-z.
• Must contain at least one numeral 0-9.
• Must contain at least one special character
• Must be longer than 6 characters. (So >= 7)
• Must be shorter than 9 characters. (So…7 or 8, but not 9)
• Must begin and end with an alpha character A-Z or a-z.
• The change periods (how often you must reset) vary.
• You may not use any password you have used in the last year.

Seriously, did they just check every available box in the security setup? They may think they’re making things more secure, but in fact, the addition of all these options, especially with the addition of the very restrictive length requirement (7 or 8 characters, really?) conspires to drastically reduce security.

It reminded me a lot of this XKCD comic:

When you take social psychology into account, you can pretty much bet the farm on the following:

• The requirements to change passwords several times a year and never repeat passwords in one year means that the month and date have to be in there.At the time of this writing, it is November 2011, so if I were trying to break a password on this system I could be reasonably sure that the password contains either 1111, 1011, 1110, 0911, 1109. That’s 5 possibilities.
• It’s pretty safe to assume the date-based string of 4 characters will appear at the end of the password, but the fact that an alpha is required at the end means users will back it up one character. Therefore the password probably fits the regex [0-9]{4}[A-Za-z]$. The addition of the letter (26 * 2 for caps) possibilities means there are only 26*2*5 = 260 likely possibilities for the last 5 characters of the password.
• If we assume 8 character passwords (would be stronger than 7 after all) then there are 3 characters left. It wouldn’t be ridiculous to assume:
• First character caps: 26 possibilities
• Second character lowercase: 26 possibilities
• Third character a symbol easily reachable from a Shift+NumberKey sequence (there are 10) plus I’ll throw in a few more for good measure that are accessible by the right pinky finger.  Let’s say 20 possibilities.
• Total possibilities for the first 3 chars = 26 * 26 * 20 = 13,520
• Total likely passwords to attempt = 13,250 * 260 = 3,515,200.

3.5 million possibilities.  Using XKCD’s assumption of 1000 guesses/second, that’s less than an hour! I sure hope they have some lockout routines on top of that password policy. Considering that they must have checked every box, I suppose I can assume they did.

You may disagree with my math or my assumptions, but that the point is that adding additional security requirements doesn’t always increase security. So get it out of your head that your users are going to pick truly random passwords and think about how they are likely to act before you consider your system to be secure.

A common question about NServiceBus is how to use it to integrate with an external partner. The requirements usually go something like this:

• The third party will contact us via a web service, passing us a transaction identifier and a collection of fields.
• If we successfully receive the message in the web service, we respond with a HTTP 200 OK status code.  If they do not receive the acknowledgement, they will assume a failure and attempt to retry the web service later.
• Once we receive the message from the third party, we need to distribute (think publish) the contents of the message to more than one internal process, each of which are completely independent of each other.
• We need to logically receive each message once and only once. In other words, it would be a “Very Bad Thing” for one of the internal subscribing processes to receive the same notification more than once.

This was most recently asked in this StackOverflow question, where it became difficult to explain more within the 600 character comment limit. The best explanation is example code, so here it is.

Check out NServiceBus External WebService Example on GitHub. Here is a high-level overview of the project:

• WebServiceHost
  • This project implements a simple ASMX web service.
  • The web request information is translated into an NServiceBus command message, and then Sent on the Bus.
• TestClient
• This console app project tests invoking the web service using a standard .NET web service proxy. No NServiceBus to be found here.
• InternalService
• An NServiceBus endpoint containing a Saga that receives the NServiceBus message from the web service.
• Even if the web service received the message successfully, we can’t know that our partner’s server didn’t fail before they received, or were able to record the acknowledgement, or that a network failure didn’t prevent our reply from arriving at all.
• Because of this, it’s possible that our partner may retry sending the message even though we’ve already received it once.  This means we may receive duplicate messages, and we need to insulate our internal processes from that.
• To do that, we accept and publish an event corresponding to the first message received.  We also store the fact that we received that message in saga data so that we will know to ignore any duplicate messages.
• We also request a timeout notification from the Timeout Manager so that after some reasonable period (after we know the partner could no longer possibly be retrying) we can clean up the saga data.
• InternalService.Messages
• This assembly contains all the message schema for the project, including:
• ExternalServiceMsg – the web service sends this to InternalService.
• IExternalMessageReceivedEvent – the event that is published upon receipt of a non-duplicated ExternalServiceMsg. We are using an interface to define the event, which is recommended as it enables easier versioning later on thanks to an interface’s multiple inheritance abilities.
• ExternalServiceSagaData – this defines the state data our saga will use to keep track of which messages it has received.
• Note that it is probably NOT best practice to keep all these things in the same assembly.  Udi would probably recommend that the command and saga data (which are internal to the logical service) be segregated from the event (which forms the external contract for the service).
• Subscriber1 and Subscriber2
• These endpoints subscribe to the IExternalMessageReceivedEvent and pump some info to the Console so that we can watch it happen.
• ExampleTimeoutManager
• For active development, you should probably just keep a Timeout Manager running as a service on your development machine, but this is provided to keep the example self-contained and as an example of how to configure the Timeout Manager package available from NuGet.  I used a fairly nonstandard queue name so that it won’t conflict if you do already have a running timeout manager on your system.

The project uses the following NuGet packages to make it as easy as possible to get started:

• Log4Net, as a dependency of NServiceBus.
• NServiceBus – for the core NServiceBus DLLs.
• A messages assembly, however, doesn’t need NServiceBus.Core.dll or Log4Net.dll, so you can just “Install-Package NServiceBus” on this assembly and then manually remove those two references.
• NServiceBus.Host – for the InternalService endpoint.
• NServiceBus.TimeoutManager

The project also uses the NuGetPowerTools package to automatically download all the required packages when you build the solution as described in this article from David Ebbo.  I highly recommend it.

For a long time I thought that the addition of a “.com” button on the iOS keyboard was a fantastic idea.  But what about .net, .org, and .edu domains?  Where’s the love for them?

I got to thinking, wouldn’t it be awesome if you could hover over the .com button and get a popup with options for the other common top-level domains?

Turns out the Apple engineers were way ahead of me.  Give it a shot sometime!

On my iPad (set to English/US keyboard layout of course) hovering over the .com button gives me additional options for .net, .org, .us, and .edu.  It does make me wonder if I would get something like .co.uk if I had a British keyboard setting.

What you apparently can NOT do is take a screenshot with the popup menu activated.

Are you experiencing either of these exceptions?

System.Security.Cryptography.CryptographicException
Padding is invalid and cannot be removed.

System.IndexOutOfRangeException
Index was outside the bounds of the array.

System.IndexOutOfRangeException
Probable I/O race condition detected while copying memory. The I/O package is not thread safe by default. In multithreaded applications, a stream must be accessed in a thread-safe way, such as a thread-safe wrapper returned by TextReader’s or TextWriter’s Synchronized methods. This also applies to classes like StreamWriter and StreamReader.

Well at least the last one is descriptive, but that is the LEAST likely to occur.

If you’re seeing any of these exceptions, there’s a good chance you’ve run afoul of a secret of the System.Cryptography namespace.  Almost nothing is thread safe.

It’s an easy error to make. We know encryption is processor intensive, and it seems like it would be smart to incur the costs of setting up ICryptoTransform objects for encryptors and decryptors once and then store them in a static variable. Any state they might share would seem to be reference data like keys and salt and init vectors, so as long as we use a new CryptoStream for each operation, what could go wrong?

Well, lots.

Internally, the implementations of ICryptoTransform (and I assume other objects) use objects from the System.IO namespace like buffers and streams that we would never think of sharing between threads, but it’s hard to know that from a simple call to ICryptoTransform.TransformBlock().

So, if you run into any of the exceptions above, try either creating your System.Cryptography objects each time you need them, or mark them with the ThreadStaticAttribute.  Remember that with [ThreadStatic], a static initializer will not execute for each thread, so check it for null before you use it, then initialize if null.

There are two types of people in the world right now: those who are angry at Netflix and those who don’t have Netflix.

Like everyone else, I received the email yesterday notifying me that as of September 1, 2011, my $10 Netflix plan that includes 1 DVD at a time and online unlimited streaming will be discontinued. Instead, they offer separate plans for DVDs and for streaming.  1 DVD at a time will now cost $8, and unlimited streaming will now also cost $8.  There is no discount for bundling, so if I want to continue the same level of service, it will now cost me $16 per month.

It’s not the money that bothers me. Prices were bound to go up.  Maybe this is a pretty severe jump all at once, but it’s not completely unexpected.

What bothers me is the false choice it represents. If money does indeed talk (and I believe it does) then Netflix is asking me to choose from these options:

1. I like getting DVDs from you, but I don’t care for your streaming service. Please take my money and keep the DVDs coming.
2. I love your streaming service, but DVDs in the mail is so 2003. Please take my money and let me stream to my heart’s content, but don’t make me walk out to the mailbox.
3. I like DVDs and I also like streaming, and I’m willing to pay more money for both.
4. Netflix, you suck. Cancel my subscription.

I don’t believe that any of these four options correctly captures my real intent:

I would be willing to pay $16 per month, maybe even more, just for the streaming service, provided that the streaming selection didn’t suck.

The streaming selection Netflix (or any other provider) has to offer is simply unacceptable.  When I started Netflix I was impressed by the streaming selection under the assumption that as time progressed, its size would only increase. But that has not been the case. As licensing contracts come and go, some movies appear and disappear from my online queue, but by and large, my opinion of the offerings as a whole has always been a resounding “Meh”.

There aren’t a lot of other market options either.  Let’s review some alternatives to Netflix:

• Amazon Prime – Cheaper than Netflix at $79/year, but all indications are that their selection is even worse than Netflix Streaming. Plus I’d need to buy a Roku box because my Blu-Ray player only has Netflix built in.
• Blockbuster – the whole appeal of Blockbuster’s DVD program was that you could return it to a store and get a new DVD faster than Netflix’s mail turnaround. With nearly every Blockbuster in my area closing in the wake of their bankruptcy filing (I honestly don’t know of a location nearby that’s still open) this advantage is moot. Fail.
• Hulu – seems to be focused more on competing with Dish and Cable than with movies. Maybe they offer some hope for the future though.
• Redbox – honestly I can see the potential appeal, but it’s never in the right place at the right time. I see people lined up at the Redbox at my local supermarket like hamsters trying to get at a feeder bar, but I want to make my entertainment decisions on my couch, not while I’m buying groceries.
• Apple TV – perhaps with an Apple TV I could rent the movies I really want to watch (starting at $3.99 each, which beats the old bricks and mortar Blockbuster price or Dish Network’s Pay-Per-View price, and can be initiated from my couch) but its other feature seem pretty limited in comparison to a Roku box.

Of course the problem isn’t Netflix at all.  Netflix can only stream what they have licensing rights to.  Unfortunately, it seems Hollywood is repeating all the mistakes of the music industry a decade ago:

Right now, in fact, the movie and TV business looks a lot like the music one did in the early 2000s. And as we’ve seen, that decade didn’t work out too well for the labels. So it’s worth looking at the situation and wondering how things are going to fare in the TV and movie world in the decade ahead.

[…]

…the music industry, when it began to feel the effects of the technological change coming, doubled down on stupid.

And so it seems the prospects of being able to stream any movie, including new releases, at any time, even if we’re willing to pay for it, are nil.

I still don’t know what I will do with my Netflix account come September 1. I want the streaming service to be great and succeed, but I’m not sure I can pump more money into a broken status quo.

Until Hollywood gets over itself, it seems I’m stuck between Netflix and a hard place.

It’s commonly very difficult to question business people about reporting requirements.  It’s not really their fault either – they just can’t know exactly what they want until they’re trying to answer a question and can’t easily do it with the reports you’ve given them.

This is why it’s good to make reports as flexible and updateable as possible, but with as little developer required to update the reports as possible.

If you’re operating in an environment where all database access must be via stored procedures, this is a really big problem.  It’s really unlikely that the changes requested by business can be implemented with the same stored procedure you naïvely created for your first attempt.  I’ve seen scenarios where a database has stored procedures with the suffixes GetReport, GetReport2, GetReport3, GetReport4, etc.  Yuck.

Even if you’re using Entity Framework, LINQ to SQL, or some other data layer framework that enables more free-form access to the database, you can’t always ensure that all report queries will result in good execution costs and actually be performant.

This is why it can sometimes be a good idea to perform a very basic database query (via stored procedure if necessary) to get a base set of data, and then perform more conditional operations on it in memory with LINQ.  It’s a pain to do a “Name Contains” filter in a stored procedure (especially if there are a dozen other options) but with LINQ it’s no big deal.

IEnumerable<DataItem> data = GetBaseData();

if (!String.IsNullOrEmpty(nameFilter))
	data = data.Where(d => d.Name.IndexOf(nameFilter, StringComparison.OrdinalIgnoreCase) >= 0);

This is really great for simple filters, but gets difficult when we want to do more complex grouping and aggregating functions, such as grouping by Hourly/Daily/Weekly/Monthly and/or by other data points.

The remainder of this article will show how this can be done with static code, and then how we can drastically increase the maintainability of this same code by employing the dynamic keyword introduced in C# 4.0.

The .NET 3.5 Way

Let’s consider the following data type.  It represents a simplified sort of reporting data you might see from an ad platform that is utilized on the web and also on mobile devices (iOS and Android).

public class ReportData
{
	public DateTime Date { get; set; }
	public string Platform { get; set; }
	public string Size { get; set; }
	public int Views { get; set; }
	public int Clicks { get; set; }
}

We want the ability to do any of these things independently or together:

• Group Date by Hour, Day, Week, Hour, or an overall Summary
• Group all platforms together
• Group all sizes together

Of course, no matter what we do, we will always sum the Views and Clicks.

Let’s start with some code to generate quite a bit of random data and then show it to us:

class Program
{
	static void Main(string[] args)
	{
		List<ReportData> data = CreateRandomData();
		Report("Raw Data, Top 10", data.Take(10));
		Console.ReadLine();
	}

	private static List<ReportData> CreateRandomData()
	{
		List<ReportData> data = new List<ReportData>();
		Random rand = new Random();
		DateTime start = new DateTime(2011,1,1);
		DateTime end = new DateTime(2011, 7, 1);

		for (DateTime dt = start; dt < end; dt = dt.AddHours(1))
		{
			foreach (string platform in new string[] { "Web", "iOS", "Android" })
			{
				foreach (string size in new string[] { "Standard", "Enhanced" })
				{
					int views = rand.Next(0, 50);
					int clicks = rand.Next(0, views / 3);
					data.Add(new ReportData
					{
						Date = dt,
						Platform = platform,
						Size = size,
						Views = views,
						Clicks = clicks
					});
				}
			}
		}

		Console.WriteLine("Generated {0} Sample Rows", data.Count);
		return data;
	}

	private static void Report(string label, IEnumerable<ReportData> data)
	{
		Console.WriteLine();
		Console.WriteLine(label);
		string format = "{0:MM/dd/yyyy HH:mm}   {1,-8}  {2,-8}  {3,6}  {4,6}";
		Console.WriteLine(format, "Date            ", "Platform", "Size", "Views", "Clicks");
		foreach (ReportData d in data)
			Console.WriteLine(format,d.Date, d.Platform, d.Size, d.Views, d.Clicks);
	}
}

This results in the following output.  (The Views and Clicks will be different each time because they are generated randomly.)

Generated 26064 Sample Rows

Raw Data, Top 10
Date               Platform  Size       Views  Clicks
01/01/2011 00:00   Web       Standard      15       2
01/01/2011 00:00   Web       Enhanced      34       1
01/01/2011 00:00   iOS       Standard      36       2
01/01/2011 00:00   iOS       Enhanced      20       5
01/01/2011 00:00   Android   Standard      44       8
01/01/2011 00:00   Android   Enhanced      29       4
01/01/2011 01:00   Web       Standard      45       9
01/01/2011 01:00   Web       Enhanced      30       7
01/01/2011 01:00   iOS       Standard      25       5
01/01/2011 01:00   iOS       Enhanced      47       9

So how would we do this filtering statically?  For each item we want to group by, we have to do a combination GroupBy-Select that groups on everything ELSE.

Here is the example for Platform and Size, which are a lot simpler than for Date.

private static IEnumerable<ReportData> ProcessVersion1(IEnumerable<ReportData> data, DateMode dateMode, bool platform, bool size)
{
	if(platform)
	{
		data = data.GroupBy(d => new
		{
			Date = d.Date,
			Size = d.Size,
		})
		.Select(g => new ReportData
		{
			Date = g.Key.Date,
			Platform = "All",
			Size = g.Key.Size,
			Views = g.Sum(d => d.Views),
			Clicks = g.Sum(d => d.Clicks)
		});
	}

	if (size)
	{
		data = data.GroupBy(d => new
		{
			Date = d.Date,
			Platform = d.Platform,
		})
		.Select(g => new ReportData
		{
			Date = g.Key.Date,
			Platform = g.Key.Platform,
			Size = "All",
			Views = g.Sum(d => d.Views),
			Clicks = g.Sum(d => d.Clicks)
		});
	}

	// Date to come later
}

So to group all the Platforms together, we must group by Date and Size, then emit new data items that have “All” for the Platform, emit the Date and Size from the grouping key, and sum the Views and Clicks.  To group all Sizes together, we must group by Date and Platform, then emit new data items that have “All” for the Size, emit the Date and Platform from the grouping key, and sum the Views and Clicks.

In short, to group an axis together, you must (for the most part) focus on all of the OTHER axes.

It gets worse for a Date grouping:

public enum DateMode
{
	Hourly,
	Daily,
	Weekly,
	Monthly,
	Summary,
}

private static IEnumerable<ReportData> ProcessVersion1(IEnumerable<ReportData> data, DateMode dateMode, bool platform, bool size)
{
	// Platform and Size, handled previously

	switch (dateMode)
	{
		case DateMode.Hourly:
			// No need to modify the data, we're assuming it comes from
			// the database already grouped by Hour
			break;
		case DateMode.Daily:
			data = data.GroupBy(d => new
			{
				Date = d.Date.Date,
				Platform = d.Platform,
				Size = d.Size
			})
			.Select(g => new ReportData
			{
				Date = g.Key.Date,
				Platform = g.Key.Platform,
				Size = g.Key.Size,
				Views = g.Sum(d => d.Views),
				Clicks = g.Sum(d => d.Clicks)
			});
			break;
		case DateMode.Weekly:
			data = data.GroupBy(d => new
			{
				Date = d.Date.Date.AddDays(-((int)d.Date.Date.DayOfWeek)),
				Platform = d.Platform,
				Size = d.Size
			})
			.Select(g => new ReportData
			{
				Date = g.Key.Date,
				Platform = g.Key.Platform,
				Size = g.Key.Size,
				Views = g.Sum(d => d.Views),
				Clicks = g.Sum(d => d.Clicks)
			});
			break;
		case DateMode.Monthly:
			data = data.GroupBy(d => new
			{
				Date = new DateTime(d.Date.Year, d.Date.Month, 1),
				Platform = d.Platform,
				Size = d.Size
			})
			.Select(g => new ReportData
			{
				Date = g.Key.Date,
				Platform = g.Key.Platform,
				Size = g.Key.Size,
				Views = g.Sum(d => d.Views),
				Clicks = g.Sum(d => d.Clicks)
			});
			break;
		case DateMode.Summary:
			data = data.GroupBy(d => new
			{
				Platform = d.Platform,
				Size = d.Size
			})
			.Select(g => new ReportData
			{
				Date = DateTime.MinValue,
				Platform = g.Key.Platform,
				Size = g.Key.Size,
				Views = g.Sum(d => d.Views),
				Clicks = g.Sum(d => d.Clicks)
			});
			break;

	}

	return data;
}

Can you imagine how much of a mess it would be if we had to add a new property to this mix?  If we were to add another property that was similar to Platform or Size, we would need to reference it twice for each of the 4 date cases (in the GroupBy and in the Select) and then twice each for the existing Platform and Size groupings, not to mention its own grouping logic.  That’s 12 edit points needed to add one property!

That simply is not maintainable, and there must be a better way.

One answer to the Stack Overflow question Dynamic LINQ GroupBy Multiple Columns suggests a way that would make this still possible without using dynamic code.  It involves making an EntryGrouper class that implements IEquatable<T> and serves as the grouping key for the GroupBy operations.  While this method should work, it is by no means short, and would have to be re-implemented for every new use case.  The use of the dynamic keyword I present can be reused over and over.

Introducing the dynamic keyword

C# 4.0 introduces the dynamic keyword (and indeed a whole new dynamic runtime) that enables us to declare any properties we want on an object, and it is not evaluated until runtime.

Now, this isn’t completely magical – you could cast any object to a dynamic and it would compile, but at runtime there must be some mechanism to provide the values and methods you try to use.

Most of the time, we will use System.Dynamic.ExpandoObject to accomplish this for us.  ExpandoObject by itself is another statically-typed object that we cannot declare random properties on.  Only when we assign it to a dynamic variable does it get this power, but under the hood, the ExpandoObject essentially stores values in a Dictionary<string, object>, and we can exploit that for our purposes.

By itself, ExpandoObject does not override GetHashCode() or Equals() from the base object, and LINQ’s GroupBy method needs this in order to do its work.  So first, we must create a wrapper that enables us to use the power of the dynamic keyword, compute equivalence and a hash code, and enable us to get back at the original data for the Select afterwards.

public class DynamicHashWrapper
{
	public dynamic Value { get; private set; }

	private Dictionary<string, object> dict;
	private HashSet<string> keys;
	private static IEqualityComparer<HashSet<string>> keyComparer = HashSet<string>.CreateSetComparer();

	public DynamicHashWrapper(dynamic value)
	{
		this.Value = value;
		dict = new Dictionary<string,object>(value as IDictionary<string, object>);
		keys = new HashSet<string>(dict.Keys);
	}

	private int? hashCode;
	public override int GetHashCode()
	{
		if (!this.hashCode.HasValue)
		{
			int code = 0;
			foreach (var entry in dict.OrderBy(p => p.Key))
				code ^= entry.Value.GetHashCode();
			this.hashCode = code;
		}
		return this.hashCode.Value;
	}

	public override bool Equals(object obj)
	{
		if (!(obj is DynamicHashWrapper))
			return false;

		DynamicHashWrapper other = obj as DynamicHashWrapper;

		return keyComparer.Equals(this.keys, other.keys)
			&& this.GetHashCode() == other.GetHashCode();
	}
}

After creating a dynamic variable, we can wrap it with this class.  It does the following:

• Stores the dynamic value itself so we can retrieve its properties back later.
• Converts the dynamic ExpandoObject to a Dictionary that we can enumerate.
• Enables computation of a hash code by XOR-ing the values of the dictionary.
• Determines if two values are equal by ensuring that the dictionary sizes are equal and that the hash codes are equal.

With this addition to the dynamic object, we can now re-implement our data processing routine:

private static IEnumerable<ReportData> ProcessVersion2(IEnumerable<ReportData> data, DateMode dateMode, bool platform, bool size)
{
	return data
		.GroupBy(d =>
		{
			dynamic key = new ExpandoObject();

			switch (dateMode)
			{
				case DateMode.Hourly: key.Date = d.Date; break;
				case DateMode.Daily: key.Date = d.Date.Date; break;
				case DateMode.Weekly: key.Date = d.Date.Date.AddDays(-((int)d.Date.DayOfWeek)); break;
				case DateMode.Monthly: key.Date = new DateTime(d.Date.Year, d.Date.Month, 1); break;
				case DateMode.Summary: key.Date = DateTime.MinValue; break;
			}

			if (!platform)
				key.Platform = d.Platform;

			if (!size)
				key.Size = d.Size;

			return new DynamicHashWrapper(key);
		})
		.Select(g => new ReportData
		{
			Date = g.Key.Value.Date,
			Platform = platform ? "All" : g.Key.Value.Platform,
			Size = size ? "All" : g.Key.Value.Size,
			Views = g.Sum(d => d.Views),
			Clicks = g.Sum(d => d.Clicks)
		});
}

Now we have a single GroupBy and a single Select.  By the way, grouping and selecting only once means we should get a performance improvement because we aren’t generating a bunch of intermediate objects each time we do a grouping – we create one dynamic grouping key and spin through the collection only once.  Of course, the dynamic object probably has its own overhead from not being statically compiled, but the improvement in maintainability is WELL worth that small setback.

As a comparison, this new implementation takes 33 lines of code, whereas the original was 111 lines.  Who wants to try to maintain a 111-line method!?

Now we can try a bunch of different groupings and see them in action.

static void Main(string[] args)
{
	List<ReportData> data = CreateRandomData();
	Report("Raw Data, Top 10", data.Take(10));
	Report("Group Platform", Process(data, DateMode.Hourly, true, false).Take(10));
	Report("Group Size", Process(data, DateMode.Hourly, false, true).Take(10));
	Report("Group Platform & Size", Process(data, DateMode.Hourly, true, true).Take(10));
	Report("Group All by Day", Process(data, DateMode.Daily, true, true).Take(10));
	Report("Group All by Week", Process(data, DateMode.Weekly, true, true).Take(10));
	Report("Group All by Month", Process(data, DateMode.Monthly, true, true).Take(10));
	Report("Group All Summary", Process(data, DateMode.Summary, true, true).Take(10));
	Report("Summary By Types", Process(data, DateMode.Summary, false, false).Take(10));
	Console.ReadLine();
}

Generated 26064 Sample Rows

Raw Data, Top 10
Date               Platform  Size       Views  Clicks
01/01/2011 00:00   Web       Standard      15       3
01/01/2011 00:00   Web       Enhanced      45       5
01/01/2011 00:00   iOS       Standard       9       1
01/01/2011 00:00   iOS       Enhanced       6       1
01/01/2011 00:00   Android   Standard       0       0
01/01/2011 00:00   Android   Enhanced       7       0
01/01/2011 01:00   Web       Standard      33       7
01/01/2011 01:00   Web       Enhanced       1       0
01/01/2011 01:00   iOS       Standard      28       7
01/01/2011 01:00   iOS       Enhanced      25       5

Group Platform
Date               Platform  Size       Views  Clicks
01/01/2011 00:00   All       Standard      24       4
01/01/2011 00:00   All       Enhanced      58       6
01/01/2011 01:00   All       Standard     105      14
01/01/2011 01:00   All       Enhanced      49       8
01/01/2011 02:00   All       Standard     133      13
01/01/2011 02:00   All       Enhanced      36       6
01/01/2011 03:00   All       Standard      95      16
01/01/2011 03:00   All       Enhanced      94      25
01/01/2011 04:00   All       Standard      58      14
01/01/2011 04:00   All       Enhanced      78       3

Group Size
Date               Platform  Size       Views  Clicks
01/01/2011 00:00   Web       All           60       8
01/01/2011 00:00   iOS       All           15       2
01/01/2011 00:00   Android   All            7       0
01/01/2011 01:00   Web       All           34       7
01/01/2011 01:00   iOS       All           53      12
01/01/2011 01:00   Android   All           67       3
01/01/2011 02:00   Web       All           55       0
01/01/2011 02:00   iOS       All           42      10
01/01/2011 02:00   Android   All           72       9
01/01/2011 03:00   Web       All           53      11

Group Platform & Size
Date               Platform  Size       Views  Clicks
01/01/2011 00:00   All       All           82      10
01/01/2011 01:00   All       All          154      22
01/01/2011 02:00   All       All          169      19
01/01/2011 03:00   All       All          189      41
01/01/2011 04:00   All       All          136      17
01/01/2011 05:00   All       All          142      17
01/01/2011 06:00   All       All          150      15
01/01/2011 07:00   All       All          181      33
01/01/2011 08:00   All       All          143      22
01/01/2011 09:00   All       All          208      34

Group All by Day
Date               Platform  Size       Views  Clicks
01/01/2011 00:00   All       All         3756     552
01/02/2011 00:00   All       All         3370     479
01/03/2011 00:00   All       All         3537     503
01/04/2011 00:00   All       All         3298     469
01/05/2011 00:00   All       All         3605     573
01/06/2011 00:00   All       All         3617     518
01/07/2011 00:00   All       All         3193     450
01/08/2011 00:00   All       All         3485     513
01/09/2011 00:00   All       All         3629     504
01/10/2011 00:00   All       All         3628     533

Group All by Week
Date               Platform  Size       Views  Clicks
12/26/2010 00:00   All       All         3756     552
01/02/2011 00:00   All       All        24105    3505
01/09/2011 00:00   All       All        24585    3495
01/16/2011 00:00   All       All        24618    3553
01/23/2011 00:00   All       All        24875    3471
01/30/2011 00:00   All       All        24845    3634
02/06/2011 00:00   All       All        24528    3391
02/13/2011 00:00   All       All        24561    3467
02/20/2011 00:00   All       All        24280    3621
02/27/2011 00:00   All       All        24614    3507

Group All by Month
Date               Platform  Size       Views  Clicks
01/01/2011 00:00   All       All       109101   15593
02/01/2011 00:00   All       All        98057   14078
03/01/2011 00:00   All       All       108231   15308
04/01/2011 00:00   All       All       106527   15286
05/01/2011 00:00   All       All       110096   15742
06/01/2011 00:00   All       All       106354   14982

Group All Summary
Date               Platform  Size       Views  Clicks
01/01/0001 00:00   All       All       638366   90989

Summary By Types
Date               Platform  Size       Views  Clicks
01/01/0001 00:00   Web       Standard  105615   14983
01/01/0001 00:00   Web       Enhanced  107740   15448
01/01/0001 00:00   iOS       Standard  107203   15417
01/01/0001 00:00   iOS       Enhanced  106457   15117
01/01/0001 00:00   Android   Standard  104772   14840
01/01/0001 00:00   Android   Enhanced  106579   15184

Summary and Code Download

With the static alternatives, we get a lot of spaghetti code that is impossible to maintain, and even if it was, the many GroupBy/Select iterations required for multiple axes result in unnecessary object creation, loops through the data collection, and eventual garbage collection.

Using the dynamic keyword and a small, reusable trick to enable the dynamic object to generate meaningful hash codes, we can loop through the data collection once.  Our results are just as good, and the next time someone visits this code, they’re much less likely to screw it up.

The code in this article is fairly fragmented. Click here to download the full source code.