I love to write code and I’m a great typist, but I hate typing if it’s not necessary. I’m a big fan of syntactical sugar and I think it’s high time C# got some more of it.

By now we’re all used to seeing this:

public string MyProperty { get; set; }

This eliminates the need to separately declare a private member variable and public property, because the compiler takes care of it for you.

What it doesn’t do is give you the option to set the initial value, like you could with a private member variable. Instead we have to do this:

public class MyClass
{
    public string MyProperty { get; set; }

    public MyClass()
    {
        MyProperty = "initial value";
    }
}

This really becomes more of a pain point now that I have delved into NServiceBus and RavenDB. With NServiceBus, I create a lot of message classes, and when the message includes a collection, it’s usually a good idea to initialize the empty collection so that I don’t get annoying null reference exceptions when I try to use them. The same is true of RavenDB when creating objects with collections to persist to the document store.

What if we could do this?

public string MyString { get; set; default "initial value"; }

public List<int> MyList { get; set; default new List<int>(); }

It would personally save me a LOT of time.

Every year users are conducting more of their lives online, but they’re also doing so from more screens than ever before. It’s not uncommon for me to access the same web application from my work computer, home computer, iPhone, and iPad, and those are just the devices that I consider “mine”. (Not that I own my work computer, but that I am its only user.)

If your web application contains anything of real value, the real danger comes from public computers that your users may use. It’s all too easy for a user to forget to log off from a website, and most webapps make it impossible to log off those sessions remotely.

I believe it’s becoming more important to allow your users the ability to globally log off from your website, clearing not only local credentials stored in cookies, but invalidating all the user’s sessions across all devices.

Some websites have already begun to do this:

• Facebook allows you to remotely log off from other devices, although the functionality is hidden within the Security area under Account Settings, at least it is today. Facebook also shows the approximate location (based on IP) of the activity, as well as a device type. Sometimes this device type is something my mom could understand (Firefox on WinVista, Facebook for iPhone) but in other cases it displays a full browser user agent, which I believe is not helpful for the vast majority of Facebook’s users.  While that is nice for me as a geek, it’s probably overkill for most applications.
• Stack Overflow (and the entire Stack Exchange network of websites) say on their Log Out page that “Clicking Log Out will clear all local credentials in your browser, and log you out on all devices.”

So how to implement global log off?

Solution 1: Session Table

This solution would look a little like the Facebook example.

When the user logs in, write a session row to a database, including the User ID and Session ID. This could include the IP address, location, and user agent information like the Facebook example, but this is not required. Write a session cookie to the user’s browser containing Session ID, salted and encrypted of course.

When the user logs off, delete all session rows for the User ID in order to effect a global log off.

This solution has the advantage that you can support disabling individual sessions without killing off all of them.

Solution 2: User Generation

A simpler solution is to add a Generation concept to your User information. This could be an integer (where every user starts at zero) or a Guid; it really doesn’t matter.

The user cookie should now include both the User ID and the Generation, again salted and encrypted.

Presumably every page request will use a (cached) User object of some type to render the obligatory “Welcome, David” user management header near the top of the page, if nothing else. When fetching the current user, the User object would be loaded by UserID, and then the Generation from the cookie is compared to the Generation in the object. If they match, the login is successful. If not, the cookie is invalid and is discarded.

When the user wants to globally log off, simply increment the Generation on the user object, and then all cookies referencing that User ID will be rendered invalid.

I have a friend.  Let’s call him “Steve”.

Steve was recently complaining to me about the password requirements imposed on him by Corporate IT.  He was using all sorts of words to describe them.  The only one suitable for reprinting was “stupid”.  They went downhill from there rather quickly.

Here are the password requirements Steve has to live with:

• Must contain at least one uppercase letter A-Z.
• Must contain at least one lowercase letter a-z.
• Must contain at least one numeral 0-9.
• Must contain at least one special character
• Must be longer than 6 characters. (So >= 7)
• Must be shorter than 9 characters. (So…7 or 8, but not 9)
• Must begin and end with an alpha character A-Z or a-z.
• The change periods (how often you must reset) vary.
• You may not use any password you have used in the last year.

Seriously, did they just check every available box in the security setup? They may think they’re making things more secure, but in fact, the addition of all these options, especially with the addition of the very restrictive length requirement (7 or 8 characters, really?) conspires to drastically reduce security.

It reminded me a lot of this XKCD comic:

When you take social psychology into account, you can pretty much bet the farm on the following:

• The requirements to change passwords several times a year and never repeat passwords in one year means that the month and date have to be in there.At the time of this writing, it is November 2011, so if I were trying to break a password on this system I could be reasonably sure that the password contains either 1111, 1011, 1110, 0911, 1109. That’s 5 possibilities.
• It’s pretty safe to assume the date-based string of 4 characters will appear at the end of the password, but the fact that an alpha is required at the end means users will back it up one character. Therefore the password probably fits the regex [0-9]{4}[A-Za-z]$. The addition of the letter (26 * 2 for caps) possibilities means there are only 26*2*5 = 260 likely possibilities for the last 5 characters of the password.
• If we assume 8 character passwords (would be stronger than 7 after all) then there are 3 characters left. It wouldn’t be ridiculous to assume:
• First character caps: 26 possibilities
• Second character lowercase: 26 possibilities
• Third character a symbol easily reachable from a Shift+NumberKey sequence (there are 10) plus I’ll throw in a few more for good measure that are accessible by the right pinky finger.  Let’s say 20 possibilities.
• Total possibilities for the first 3 chars = 26 * 26 * 20 = 13,520
• Total likely passwords to attempt = 13,250 * 260 = 3,515,200.

3.5 million possibilities.  Using XKCD’s assumption of 1000 guesses/second, that’s less than an hour! I sure hope they have some lockout routines on top of that password policy. Considering that they must have checked every box, I suppose I can assume they did.

You may disagree with my math or my assumptions, but that the point is that adding additional security requirements doesn’t always increase security. So get it out of your head that your users are going to pick truly random passwords and think about how they are likely to act before you consider your system to be secure.

A common question about NServiceBus is how to use it to integrate with an external partner. The requirements usually go something like this:

• The third party will contact us via a web service, passing us a transaction identifier and a collection of fields.
• If we successfully receive the message in the web service, we respond with a HTTP 200 OK status code.  If they do not receive the acknowledgement, they will assume a failure and attempt to retry the web service later.
• Once we receive the message from the third party, we need to distribute (think publish) the contents of the message to more than one internal process, each of which are completely independent of each other.
• We need to logically receive each message once and only once. In other words, it would be a “Very Bad Thing” for one of the internal subscribing processes to receive the same notification more than once.

This was most recently asked in this StackOverflow question, where it became difficult to explain more within the 600 character comment limit. The best explanation is example code, so here it is.

Check out NServiceBus External WebService Example on GitHub. Here is a high-level overview of the project:

• WebServiceHost
  • This project implements a simple ASMX web service.
  • The web request information is translated into an NServiceBus command message, and then Sent on the Bus.
• TestClient
• This console app project tests invoking the web service using a standard .NET web service proxy. No NServiceBus to be found here.
• InternalService
• An NServiceBus endpoint containing a Saga that receives the NServiceBus message from the web service.
• Even if the web service received the message successfully, we can’t know that our partner’s server didn’t fail before they received, or were able to record the acknowledgement, or that a network failure didn’t prevent our reply from arriving at all.
• Because of this, it’s possible that our partner may retry sending the message even though we’ve already received it once.  This means we may receive duplicate messages, and we need to insulate our internal processes from that.
• To do that, we accept and publish an event corresponding to the first message received.  We also store the fact that we received that message in saga data so that we will know to ignore any duplicate messages.
• We also request a timeout notification from the Timeout Manager so that after some reasonable period (after we know the partner could no longer possibly be retrying) we can clean up the saga data.
• InternalService.Messages
• This assembly contains all the message schema for the project, including:
• ExternalServiceMsg – the web service sends this to InternalService.
• IExternalMessageReceivedEvent – the event that is published upon receipt of a non-duplicated ExternalServiceMsg. We are using an interface to define the event, which is recommended as it enables easier versioning later on thanks to an interface’s multiple inheritance abilities.
• ExternalServiceSagaData – this defines the state data our saga will use to keep track of which messages it has received.
• Note that it is probably NOT best practice to keep all these things in the same assembly.  Udi would probably recommend that the command and saga data (which are internal to the logical service) be segregated from the event (which forms the external contract for the service).
• Subscriber1 and Subscriber2
• These endpoints subscribe to the IExternalMessageReceivedEvent and pump some info to the Console so that we can watch it happen.
• ExampleTimeoutManager
• For active development, you should probably just keep a Timeout Manager running as a service on your development machine, but this is provided to keep the example self-contained and as an example of how to configure the Timeout Manager package available from NuGet.  I used a fairly nonstandard queue name so that it won’t conflict if you do already have a running timeout manager on your system.

The project uses the following NuGet packages to make it as easy as possible to get started:

• Log4Net, as a dependency of NServiceBus.
• NServiceBus – for the core NServiceBus DLLs.
• A messages assembly, however, doesn’t need NServiceBus.Core.dll or Log4Net.dll, so you can just “Install-Package NServiceBus” on this assembly and then manually remove those two references.
• NServiceBus.Host – for the InternalService endpoint.
• NServiceBus.TimeoutManager

The project also uses the NuGetPowerTools package to automatically download all the required packages when you build the solution as described in this article from David Ebbo.  I highly recommend it.

For a long time I thought that the addition of a “.com” button on the iOS keyboard was a fantastic idea.  But what about .net, .org, and .edu domains?  Where’s the love for them?

I got to thinking, wouldn’t it be awesome if you could hover over the .com button and get a popup with options for the other common top-level domains?

Turns out the Apple engineers were way ahead of me.  Give it a shot sometime!

On my iPad (set to English/US keyboard layout of course) hovering over the .com button gives me additional options for .net, .org, .us, and .edu.  It does make me wonder if I would get something like .co.uk if I had a British keyboard setting.

What you apparently can NOT do is take a screenshot with the popup menu activated.

Are you experiencing either of these exceptions?

System.Security.Cryptography.CryptographicException
Padding is invalid and cannot be removed.

System.IndexOutOfRangeException
Index was outside the bounds of the array.

System.IndexOutOfRangeException
Probable I/O race condition detected while copying memory. The I/O package is not thread safe by default. In multithreaded applications, a stream must be accessed in a thread-safe way, such as a thread-safe wrapper returned by TextReader’s or TextWriter’s Synchronized methods. This also applies to classes like StreamWriter and StreamReader.

Well at least the last one is descriptive, but that is the LEAST likely to occur.

If you’re seeing any of these exceptions, there’s a good chance you’ve run afoul of a secret of the System.Cryptography namespace.  Almost nothing is thread safe.

It’s an easy error to make. We know encryption is processor intensive, and it seems like it would be smart to incur the costs of setting up ICryptoTransform objects for encryptors and decryptors once and then store them in a static variable. Any state they might share would seem to be reference data like keys and salt and init vectors, so as long as we use a new CryptoStream for each operation, what could go wrong?

Well, lots.

Internally, the implementations of ICryptoTransform (and I assume other objects) use objects from the System.IO namespace like buffers and streams that we would never think of sharing between threads, but it’s hard to know that from a simple call to ICryptoTransform.TransformBlock().

So, if you run into any of the exceptions above, try either creating your System.Cryptography objects each time you need them, or mark them with the ThreadStaticAttribute.  Remember that with [ThreadStatic], a static initializer will not execute for each thread, so check it for null before you use it, then initialize if null.

There are two types of people in the world right now: those who are angry at Netflix and those who don’t have Netflix.

Like everyone else, I received the email yesterday notifying me that as of September 1, 2011, my $10 Netflix plan that includes 1 DVD at a time and online unlimited streaming will be discontinued. Instead, they offer separate plans for DVDs and for streaming.  1 DVD at a time will now cost $8, and unlimited streaming will now also cost $8.  There is no discount for bundling, so if I want to continue the same level of service, it will now cost me $16 per month.

It’s not the money that bothers me. Prices were bound to go up.  Maybe this is a pretty severe jump all at once, but it’s not completely unexpected.

What bothers me is the false choice it represents. If money does indeed talk (and I believe it does) then Netflix is asking me to choose from these options:

1. I like getting DVDs from you, but I don’t care for your streaming service. Please take my money and keep the DVDs coming.
2. I love your streaming service, but DVDs in the mail is so 2003. Please take my money and let me stream to my heart’s content, but don’t make me walk out to the mailbox.
3. I like DVDs and I also like streaming, and I’m willing to pay more money for both.
4. Netflix, you suck. Cancel my subscription.

I don’t believe that any of these four options correctly captures my real intent:

I would be willing to pay $16 per month, maybe even more, just for the streaming service, provided that the streaming selection didn’t suck.

The streaming selection Netflix (or any other provider) has to offer is simply unacceptable.  When I started Netflix I was impressed by the streaming selection under the assumption that as time progressed, its size would only increase. But that has not been the case. As licensing contracts come and go, some movies appear and disappear from my online queue, but by and large, my opinion of the offerings as a whole has always been a resounding “Meh”.

There aren’t a lot of other market options either.  Let’s review some alternatives to Netflix:

• Amazon Prime – Cheaper than Netflix at $79/year, but all indications are that their selection is even worse than Netflix Streaming. Plus I’d need to buy a Roku box because my Blu-Ray player only has Netflix built in.
• Blockbuster – the whole appeal of Blockbuster’s DVD program was that you could return it to a store and get a new DVD faster than Netflix’s mail turnaround. With nearly every Blockbuster in my area closing in the wake of their bankruptcy filing (I honestly don’t know of a location nearby that’s still open) this advantage is moot. Fail.
• Hulu – seems to be focused more on competing with Dish and Cable than with movies. Maybe they offer some hope for the future though.
• Redbox – honestly I can see the potential appeal, but it’s never in the right place at the right time. I see people lined up at the Redbox at my local supermarket like hamsters trying to get at a feeder bar, but I want to make my entertainment decisions on my couch, not while I’m buying groceries.
• Apple TV – perhaps with an Apple TV I could rent the movies I really want to watch (starting at $3.99 each, which beats the old bricks and mortar Blockbuster price or Dish Network’s Pay-Per-View price, and can be initiated from my couch) but its other feature seem pretty limited in comparison to a Roku box.

Of course the problem isn’t Netflix at all.  Netflix can only stream what they have licensing rights to.  Unfortunately, it seems Hollywood is repeating all the mistakes of the music industry a decade ago:

Right now, in fact, the movie and TV business looks a lot like the music one did in the early 2000s. And as we’ve seen, that decade didn’t work out too well for the labels. So it’s worth looking at the situation and wondering how things are going to fare in the TV and movie world in the decade ahead.

[…]

…the music industry, when it began to feel the effects of the technological change coming, doubled down on stupid.

And so it seems the prospects of being able to stream any movie, including new releases, at any time, even if we’re willing to pay for it, are nil.

I still don’t know what I will do with my Netflix account come September 1. I want the streaming service to be great and succeed, but I’m not sure I can pump more money into a broken status quo.

Until Hollywood gets over itself, it seems I’m stuck between Netflix and a hard place.

It’s commonly very difficult to question business people about reporting requirements.  It’s not really their fault either – they just can’t know exactly what they want until they’re trying to answer a question and can’t easily do it with the reports you’ve given them.

This is why it’s good to make reports as flexible and updateable as possible, but with as little developer required to update the reports as possible.

If you’re operating in an environment where all database access must be via stored procedures, this is a really big problem.  It’s really unlikely that the changes requested by business can be implemented with the same stored procedure you naïvely created for your first attempt.  I’ve seen scenarios where a database has stored procedures with the suffixes GetReport, GetReport2, GetReport3, GetReport4, etc.  Yuck.

Even if you’re using Entity Framework, LINQ to SQL, or some other data layer framework that enables more free-form access to the database, you can’t always ensure that all report queries will result in good execution costs and actually be performant.

This is why it can sometimes be a good idea to perform a very basic database query (via stored procedure if necessary) to get a base set of data, and then perform more conditional operations on it in memory with LINQ.  It’s a pain to do a “Name Contains” filter in a stored procedure (especially if there are a dozen other options) but with LINQ it’s no big deal.

IEnumerable<DataItem> data = GetBaseData();

if (!String.IsNullOrEmpty(nameFilter))
	data = data.Where(d => d.Name.IndexOf(nameFilter, StringComparison.OrdinalIgnoreCase) >= 0);

This is really great for simple filters, but gets difficult when we want to do more complex grouping and aggregating functions, such as grouping by Hourly/Daily/Weekly/Monthly and/or by other data points.

The remainder of this article will show how this can be done with static code, and then how we can drastically increase the maintainability of this same code by employing the dynamic keyword introduced in C# 4.0.

The .NET 3.5 Way

Let’s consider the following data type.  It represents a simplified sort of reporting data you might see from an ad platform that is utilized on the web and also on mobile devices (iOS and Android).

public class ReportData
{
	public DateTime Date { get; set; }
	public string Platform { get; set; }
	public string Size { get; set; }
	public int Views { get; set; }
	public int Clicks { get; set; }
}

We want the ability to do any of these things independently or together:

• Group Date by Hour, Day, Week, Hour, or an overall Summary
• Group all platforms together
• Group all sizes together

Of course, no matter what we do, we will always sum the Views and Clicks.

Let’s start with some code to generate quite a bit of random data and then show it to us:

class Program
{
	static void Main(string[] args)
	{
		List<ReportData> data = CreateRandomData();
		Report("Raw Data, Top 10", data.Take(10));
		Console.ReadLine();
	}

	private static List<ReportData> CreateRandomData()
	{
		List<ReportData> data = new List<ReportData>();
		Random rand = new Random();
		DateTime start = new DateTime(2011,1,1);
		DateTime end = new DateTime(2011, 7, 1);

		for (DateTime dt = start; dt < end; dt = dt.AddHours(1))
		{
			foreach (string platform in new string[] { "Web", "iOS", "Android" })
			{
				foreach (string size in new string[] { "Standard", "Enhanced" })
				{
					int views = rand.Next(0, 50);
					int clicks = rand.Next(0, views / 3);
					data.Add(new ReportData
					{
						Date = dt,
						Platform = platform,
						Size = size,
						Views = views,
						Clicks = clicks
					});
				}
			}
		}

		Console.WriteLine("Generated {0} Sample Rows", data.Count);
		return data;
	}

	private static void Report(string label, IEnumerable<ReportData> data)
	{
		Console.WriteLine();
		Console.WriteLine(label);
		string format = "{0:MM/dd/yyyy HH:mm}   {1,-8}  {2,-8}  {3,6}  {4,6}";
		Console.WriteLine(format, "Date            ", "Platform", "Size", "Views", "Clicks");
		foreach (ReportData d in data)
			Console.WriteLine(format,d.Date, d.Platform, d.Size, d.Views, d.Clicks);
	}
}

This results in the following output.  (The Views and Clicks will be different each time because they are generated randomly.)

Generated 26064 Sample Rows

Raw Data, Top 10
Date               Platform  Size       Views  Clicks
01/01/2011 00:00   Web       Standard      15       2
01/01/2011 00:00   Web       Enhanced      34       1
01/01/2011 00:00   iOS       Standard      36       2
01/01/2011 00:00   iOS       Enhanced      20       5
01/01/2011 00:00   Android   Standard      44       8
01/01/2011 00:00   Android   Enhanced      29       4
01/01/2011 01:00   Web       Standard      45       9
01/01/2011 01:00   Web       Enhanced      30       7
01/01/2011 01:00   iOS       Standard      25       5
01/01/2011 01:00   iOS       Enhanced      47       9

So how would we do this filtering statically?  For each item we want to group by, we have to do a combination GroupBy-Select that groups on everything ELSE.

Here is the example for Platform and Size, which are a lot simpler than for Date.

private static IEnumerable<ReportData> ProcessVersion1(IEnumerable<ReportData> data, DateMode dateMode, bool platform, bool size)
{
	if(platform)
	{
		data = data.GroupBy(d => new
		{
			Date = d.Date,
			Size = d.Size,
		})
		.Select(g => new ReportData
		{
			Date = g.Key.Date,
			Platform = "All",
			Size = g.Key.Size,
			Views = g.Sum(d => d.Views),
			Clicks = g.Sum(d => d.Clicks)
		});
	}

	if (size)
	{
		data = data.GroupBy(d => new
		{
			Date = d.Date,
			Platform = d.Platform,
		})
		.Select(g => new ReportData
		{
			Date = g.Key.Date,
			Platform = g.Key.Platform,
			Size = "All",
			Views = g.Sum(d => d.Views),
			Clicks = g.Sum(d => d.Clicks)
		});
	}

	// Date to come later
}

So to group all the Platforms together, we must group by Date and Size, then emit new data items that have “All” for the Platform, emit the Date and Size from the grouping key, and sum the Views and Clicks.  To group all Sizes together, we must group by Date and Platform, then emit new data items that have “All” for the Size, emit the Date and Platform from the grouping key, and sum the Views and Clicks.

In short, to group an axis together, you must (for the most part) focus on all of the OTHER axes.

It gets worse for a Date grouping:

public enum DateMode
{
	Hourly,
	Daily,
	Weekly,
	Monthly,
	Summary,
}

private static IEnumerable<ReportData> ProcessVersion1(IEnumerable<ReportData> data, DateMode dateMode, bool platform, bool size)
{
	// Platform and Size, handled previously

	switch (dateMode)
	{
		case DateMode.Hourly:
			// No need to modify the data, we're assuming it comes from
			// the database already grouped by Hour
			break;
		case DateMode.Daily:
			data = data.GroupBy(d => new
			{
				Date = d.Date.Date,
				Platform = d.Platform,
				Size = d.Size
			})
			.Select(g => new ReportData
			{
				Date = g.Key.Date,
				Platform = g.Key.Platform,
				Size = g.Key.Size,
				Views = g.Sum(d => d.Views),
				Clicks = g.Sum(d => d.Clicks)
			});
			break;
		case DateMode.Weekly:
			data = data.GroupBy(d => new
			{
				Date = d.Date.Date.AddDays(-((int)d.Date.Date.DayOfWeek)),
				Platform = d.Platform,
				Size = d.Size
			})
			.Select(g => new ReportData
			{
				Date = g.Key.Date,
				Platform = g.Key.Platform,
				Size = g.Key.Size,
				Views = g.Sum(d => d.Views),
				Clicks = g.Sum(d => d.Clicks)
			});
			break;
		case DateMode.Monthly:
			data = data.GroupBy(d => new
			{
				Date = new DateTime(d.Date.Year, d.Date.Month, 1),
				Platform = d.Platform,
				Size = d.Size
			})
			.Select(g => new ReportData
			{
				Date = g.Key.Date,
				Platform = g.Key.Platform,
				Size = g.Key.Size,
				Views = g.Sum(d => d.Views),
				Clicks = g.Sum(d => d.Clicks)
			});
			break;
		case DateMode.Summary:
			data = data.GroupBy(d => new
			{
				Platform = d.Platform,
				Size = d.Size
			})
			.Select(g => new ReportData
			{
				Date = DateTime.MinValue,
				Platform = g.Key.Platform,
				Size = g.Key.Size,
				Views = g.Sum(d => d.Views),
				Clicks = g.Sum(d => d.Clicks)
			});
			break;

	}

	return data;
}

Can you imagine how much of a mess it would be if we had to add a new property to this mix?  If we were to add another property that was similar to Platform or Size, we would need to reference it twice for each of the 4 date cases (in the GroupBy and in the Select) and then twice each for the existing Platform and Size groupings, not to mention its own grouping logic.  That’s 12 edit points needed to add one property!

That simply is not maintainable, and there must be a better way.

One answer to the Stack Overflow question Dynamic LINQ GroupBy Multiple Columns suggests a way that would make this still possible without using dynamic code.  It involves making an EntryGrouper class that implements IEquatable<T> and serves as the grouping key for the GroupBy operations.  While this method should work, it is by no means short, and would have to be re-implemented for every new use case.  The use of the dynamic keyword I present can be reused over and over.

Introducing the dynamic keyword

C# 4.0 introduces the dynamic keyword (and indeed a whole new dynamic runtime) that enables us to declare any properties we want on an object, and it is not evaluated until runtime.

Now, this isn’t completely magical – you could cast any object to a dynamic and it would compile, but at runtime there must be some mechanism to provide the values and methods you try to use.

Most of the time, we will use System.Dynamic.ExpandoObject to accomplish this for us.  ExpandoObject by itself is another statically-typed object that we cannot declare random properties on.  Only when we assign it to a dynamic variable does it get this power, but under the hood, the ExpandoObject essentially stores values in a Dictionary<string, object>, and we can exploit that for our purposes.

By itself, ExpandoObject does not override GetHashCode() or Equals() from the base object, and LINQ’s GroupBy method needs this in order to do its work.  So first, we must create a wrapper that enables us to use the power of the dynamic keyword, compute equivalence and a hash code, and enable us to get back at the original data for the Select afterwards.

public class DynamicHashWrapper
{
	public dynamic Value { get; private set; }

	private Dictionary<string, object> dict;
	private HashSet<string> keys;
	private static IEqualityComparer<HashSet<string>> keyComparer = HashSet<string>.CreateSetComparer();

	public DynamicHashWrapper(dynamic value)
	{
		this.Value = value;
		dict = new Dictionary<string,object>(value as IDictionary<string, object>);
		keys = new HashSet<string>(dict.Keys);
	}

	private int? hashCode;
	public override int GetHashCode()
	{
		if (!this.hashCode.HasValue)
		{
			int code = 0;
			foreach (var entry in dict.OrderBy(p => p.Key))
				code ^= entry.Value.GetHashCode();
			this.hashCode = code;
		}
		return this.hashCode.Value;
	}

	public override bool Equals(object obj)
	{
		if (!(obj is DynamicHashWrapper))
			return false;

		DynamicHashWrapper other = obj as DynamicHashWrapper;

		return keyComparer.Equals(this.keys, other.keys)
			&& this.GetHashCode() == other.GetHashCode();
	}
}

After creating a dynamic variable, we can wrap it with this class.  It does the following:

• Stores the dynamic value itself so we can retrieve its properties back later.
• Converts the dynamic ExpandoObject to a Dictionary that we can enumerate.
• Enables computation of a hash code by XOR-ing the values of the dictionary.
• Determines if two values are equal by ensuring that the dictionary sizes are equal and that the hash codes are equal.

With this addition to the dynamic object, we can now re-implement our data processing routine:

private static IEnumerable<ReportData> ProcessVersion2(IEnumerable<ReportData> data, DateMode dateMode, bool platform, bool size)
{
	return data
		.GroupBy(d =>
		{
			dynamic key = new ExpandoObject();

			switch (dateMode)
			{
				case DateMode.Hourly: key.Date = d.Date; break;
				case DateMode.Daily: key.Date = d.Date.Date; break;
				case DateMode.Weekly: key.Date = d.Date.Date.AddDays(-((int)d.Date.DayOfWeek)); break;
				case DateMode.Monthly: key.Date = new DateTime(d.Date.Year, d.Date.Month, 1); break;
				case DateMode.Summary: key.Date = DateTime.MinValue; break;
			}

			if (!platform)
				key.Platform = d.Platform;

			if (!size)
				key.Size = d.Size;

			return new DynamicHashWrapper(key);
		})
		.Select(g => new ReportData
		{
			Date = g.Key.Value.Date,
			Platform = platform ? "All" : g.Key.Value.Platform,
			Size = size ? "All" : g.Key.Value.Size,
			Views = g.Sum(d => d.Views),
			Clicks = g.Sum(d => d.Clicks)
		});
}

Now we have a single GroupBy and a single Select.  By the way, grouping and selecting only once means we should get a performance improvement because we aren’t generating a bunch of intermediate objects each time we do a grouping – we create one dynamic grouping key and spin through the collection only once.  Of course, the dynamic object probably has its own overhead from not being statically compiled, but the improvement in maintainability is WELL worth that small setback.

As a comparison, this new implementation takes 33 lines of code, whereas the original was 111 lines.  Who wants to try to maintain a 111-line method!?

Now we can try a bunch of different groupings and see them in action.

static void Main(string[] args)
{
	List<ReportData> data = CreateRandomData();
	Report("Raw Data, Top 10", data.Take(10));
	Report("Group Platform", Process(data, DateMode.Hourly, true, false).Take(10));
	Report("Group Size", Process(data, DateMode.Hourly, false, true).Take(10));
	Report("Group Platform & Size", Process(data, DateMode.Hourly, true, true).Take(10));
	Report("Group All by Day", Process(data, DateMode.Daily, true, true).Take(10));
	Report("Group All by Week", Process(data, DateMode.Weekly, true, true).Take(10));
	Report("Group All by Month", Process(data, DateMode.Monthly, true, true).Take(10));
	Report("Group All Summary", Process(data, DateMode.Summary, true, true).Take(10));
	Report("Summary By Types", Process(data, DateMode.Summary, false, false).Take(10));
	Console.ReadLine();
}

Generated 26064 Sample Rows

Raw Data, Top 10
Date               Platform  Size       Views  Clicks
01/01/2011 00:00   Web       Standard      15       3
01/01/2011 00:00   Web       Enhanced      45       5
01/01/2011 00:00   iOS       Standard       9       1
01/01/2011 00:00   iOS       Enhanced       6       1
01/01/2011 00:00   Android   Standard       0       0
01/01/2011 00:00   Android   Enhanced       7       0
01/01/2011 01:00   Web       Standard      33       7
01/01/2011 01:00   Web       Enhanced       1       0
01/01/2011 01:00   iOS       Standard      28       7
01/01/2011 01:00   iOS       Enhanced      25       5

Group Platform
Date               Platform  Size       Views  Clicks
01/01/2011 00:00   All       Standard      24       4
01/01/2011 00:00   All       Enhanced      58       6
01/01/2011 01:00   All       Standard     105      14
01/01/2011 01:00   All       Enhanced      49       8
01/01/2011 02:00   All       Standard     133      13
01/01/2011 02:00   All       Enhanced      36       6
01/01/2011 03:00   All       Standard      95      16
01/01/2011 03:00   All       Enhanced      94      25
01/01/2011 04:00   All       Standard      58      14
01/01/2011 04:00   All       Enhanced      78       3

Group Size
Date               Platform  Size       Views  Clicks
01/01/2011 00:00   Web       All           60       8
01/01/2011 00:00   iOS       All           15       2
01/01/2011 00:00   Android   All            7       0
01/01/2011 01:00   Web       All           34       7
01/01/2011 01:00   iOS       All           53      12
01/01/2011 01:00   Android   All           67       3
01/01/2011 02:00   Web       All           55       0
01/01/2011 02:00   iOS       All           42      10
01/01/2011 02:00   Android   All           72       9
01/01/2011 03:00   Web       All           53      11

Group Platform & Size
Date               Platform  Size       Views  Clicks
01/01/2011 00:00   All       All           82      10
01/01/2011 01:00   All       All          154      22
01/01/2011 02:00   All       All          169      19
01/01/2011 03:00   All       All          189      41
01/01/2011 04:00   All       All          136      17
01/01/2011 05:00   All       All          142      17
01/01/2011 06:00   All       All          150      15
01/01/2011 07:00   All       All          181      33
01/01/2011 08:00   All       All          143      22
01/01/2011 09:00   All       All          208      34

Group All by Day
Date               Platform  Size       Views  Clicks
01/01/2011 00:00   All       All         3756     552
01/02/2011 00:00   All       All         3370     479
01/03/2011 00:00   All       All         3537     503
01/04/2011 00:00   All       All         3298     469
01/05/2011 00:00   All       All         3605     573
01/06/2011 00:00   All       All         3617     518
01/07/2011 00:00   All       All         3193     450
01/08/2011 00:00   All       All         3485     513
01/09/2011 00:00   All       All         3629     504
01/10/2011 00:00   All       All         3628     533

Group All by Week
Date               Platform  Size       Views  Clicks
12/26/2010 00:00   All       All         3756     552
01/02/2011 00:00   All       All        24105    3505
01/09/2011 00:00   All       All        24585    3495
01/16/2011 00:00   All       All        24618    3553
01/23/2011 00:00   All       All        24875    3471
01/30/2011 00:00   All       All        24845    3634
02/06/2011 00:00   All       All        24528    3391
02/13/2011 00:00   All       All        24561    3467
02/20/2011 00:00   All       All        24280    3621
02/27/2011 00:00   All       All        24614    3507

Group All by Month
Date               Platform  Size       Views  Clicks
01/01/2011 00:00   All       All       109101   15593
02/01/2011 00:00   All       All        98057   14078
03/01/2011 00:00   All       All       108231   15308
04/01/2011 00:00   All       All       106527   15286
05/01/2011 00:00   All       All       110096   15742
06/01/2011 00:00   All       All       106354   14982

Group All Summary
Date               Platform  Size       Views  Clicks
01/01/0001 00:00   All       All       638366   90989

Summary By Types
Date               Platform  Size       Views  Clicks
01/01/0001 00:00   Web       Standard  105615   14983
01/01/0001 00:00   Web       Enhanced  107740   15448
01/01/0001 00:00   iOS       Standard  107203   15417
01/01/0001 00:00   iOS       Enhanced  106457   15117
01/01/0001 00:00   Android   Standard  104772   14840
01/01/0001 00:00   Android   Enhanced  106579   15184

Summary and Code Download

With the static alternatives, we get a lot of spaghetti code that is impossible to maintain, and even if it was, the many GroupBy/Select iterations required for multiple axes result in unnecessary object creation, loops through the data collection, and eventual garbage collection.

Using the dynamic keyword and a small, reusable trick to enable the dynamic object to generate meaningful hash codes, we can loop through the data collection once.  Our results are just as good, and the next time someone visits this code, they’re much less likely to screw it up.

The code in this article is fairly fragmented. Click here to download the full source code.

At work we use Hudson Continuous Integration for our build servers because, among other reasons:

• It’s FREE!
• It runs on Windows (for our C# builds) and on Mac/Linux (for our iOS/Android builds).
• It has a web-based GUI that is MUCH easier to use than the XML-driven config used by CruiseControl.NET, which we used before switching to Hudson.
• It has a rich system of plugins for adding functionality.
• Did I mention it’s FREE?

The one nice thing about CruiseControl.NET was that because it had one complex XML configuration file, I would only edit that file in source control so that I could back out my changes if I screwed it up. Now I need a way to back up the Hudson configuration files so that if one of my build servers goes up in flames, I can get my team back in business quickly.

A good backup solution needs to be automatic and offsite, and due to the magic of distributed version control and the inherent job execution nature of Hudson, we can back up Hudson with Hudson. If this isn’t the ultimate in universe folding in on itself awesome, I don’t know what is.

To get your own backup started:

1. Get an offsite Mercurial repository that you can push to over the Internet using the credentials that your Hudson server runs under. We use (and I heavily recommend) Kiln from Fog Creek Software, which combines hosted Mercurial source control with killer code review tools and a host of other power tools, such as a code search facility that I now could not live without. Check it out – it will change your life.
2. Make sure you have Hudson’s Mercurial plugin installed.
3. Create a new Hudson project named “Hudson Backup” or something similar.
4. Under Source Code Management, enter your Mercurial repository URL. Unlike a normal project where you pull code from a repository and then build it, we will instead be pushing backups to this repository, but defining it here identifies it as Hudson’s workspace.
5. Under Build Triggers, check “Build Periodically” and in the Schedule textbox, enter “@hourly”. This will run the backup job every hour, but will only push new changes to your Mercurial repository when there are changes (when you’ve modified something in the Web UI).
6. Under Build, add a new build step of type “Execute Windows batch command”. The script follows:

:: Copy the main Hudson config file
copy "%BASE%\config.xml" "%WORKSPACE%\config.xml"

:: Copy each job's config file
for /f "delims=|" %%f in ('dir /b %BASE%\jobs') do (
mkdir "%WORKSPACE%\jobs\%%f"
copy "%BASE%\jobs\%%f\config.xml" "%WORKSPACE%\jobs\%%f\config.xml" /Y
)

:: If there are changes, commit them to local repository then push
hg commit --addremove --message "Backing up changes to Hudson configuration."
hg push

Now, the job will run every hour, and if there are any changes, they will be committed and pushed to your remote Mercurial repository. Even your backup job is backed up by your backup job!

If you were to ever lose your build server, you could set up a new Hudson server, clone your Mercurial repository locally, copy the backup config files to your Hudson directory, and then execute the Reload Configuration from Disk command in Hudson, and you’re back in business.

When coding for work, everything of course has to be done the Right Way®. This isn’t always super exciting, so it is sometimes liberating to cut loose and work on a side project that mashes together a whole bunch of technologies without worrying too much about stability, reliability, scalability, or even if it will continue to run tomorrow. These R&D projects will never have even a single line of code directly pushed into even a development repository, but more often than not I find that I take concepts learned and tested during these coding sessions and apply them in some later project. Even if the entire project is thrown away in relatively short order, some concept of value survives for the long haul.

Plus, it’s just fun.

Recently my wife and I got the very exciting (and scary!) news that we were pregnant with our first child. The little guy or girl’s arrival is still over 5 months away, but already we’re wrestling with tons of difficult questions, and one particularly overwhelming one is “How are we going to decide where to send our child for day care?”

We live in the great state of Minnesota where the Department of Human Services maintains a searchable Licensing Info Lookup website for all sorts of things, including (but not limited to) family child care. Anyone with a child care license can be found here, along with address, phone number, if they can accept newborn infants and how many, etc.

Just one problem. We live on the border of two big suburbs, so you do a search for both cities and together you get over 150 results, and no map.

This is where my inner geek starts to get excited. I’ve got a copy of Visual Studio. I can fix this problem. Let’s do it.

Screen Scraping with YQL

Of course it’s not possible for a web page to directly access data from a different domain, but using a proxy capable of JSONP, it’s possible to grab data from any website, format it into JSON, and then surround it with a callback parameter so that you can inject it as a script tag into your document’s HEAD, where it will execute your callback function by passing the data inside. If you’re not familiar with this, check out Wikipedia’s JSONP page.

It turns out that YQL (Yahoo! Query Language) can serve as the perfect proxy in this situation. In a real-life project, I would be hesitant to rely on YQL as Yahoo could begin to reject an application for high traffic, or just pull the plug on YQL altogether. But for a low traffic site (hit by only myself and my wife) it’s a perfect match.

First you need to analyze the page you wish to scrape. On each of the two search result pages (one for each city) I wished to scrape, the HTML content for each search result looked something like this, where I’ve replaced any real content with {Placeholders}.

<table  border="0" summary="" class="LicTable1">
	<tr>
		<td width="70%" class="LicTitle1"><a href="Details.aspx?l={ResultID}" class="resultsNote">{ResultName}</a></td>
		<td  width="30%" class="LicStatus1">Active</td>
	</tr>
</table>

<table  border="0" summary="" class="LicTable">
	<tr>
		<td width="30%" class="LicContentL">{StreetAddress}<br>{CityStateZip}<br>
		{PhoneNumber}<br>
		{CountyName}</td>
		<td width="70%"  class="LicContentR">License number: {LicenseNumber}
		<br /><br /><br />Type of service: Family Child Care</td>
	</tr>
	<!-- SECTION : Optional Row -->
</table>

<img height="10" src="Images/blank.gif" alt="" />

Really, Minnesota? Table-driven design? Ever heard of semantic markup? But I digress. As much as I detest this markup (I would instantly reject it if I saw it in a code review from my own developers) it has enough detail that I can work with this.

The YQL expression goes like this:

select * from html
where url = "{Url}"
	and xpath='//table[@class="LicTable1" or @class="LicTable"]'

YQL can fetch the original HTML, perform an xpath to locate any table node with the class name “LicTable1″ or “LicTable” and then return those results. To try it for yourself, head over to the Minnesota Department of Human Services Licensing Lookup, perform any search you like, drop the search results URL into the format above, and then drop that into the YQL Console. If it doesn’t work anymore, it’s probably because the MN-DHS changed the markup on the website. That’s what you get when you try to screen scrape – nobody is under any obligation to adhere to any sort of contract in their HTML markup.

You’ll find that YQL can return its results in XML or in JSON, where in JSON the HTML markup is converted into JSON objects. If you implement the callback function (the YQL Console uses “cbfunc” by default, I’m using $.parseData) you can parse through the HTML structure shown above like this:

var data = { List: [] };

$.parseData = function (d) {
	var cur = null;
	$(d.query.results.table).each(function (i, tbl) {
		if (tbl.class == "LicTable1") {
			cur = {
				name: tbl.tr.td[0].a.content,
				href: "http://licensinglookup.dhs.state.mn.us/" + tbl.tr.td[0].a.href,
				enabled: true
			};
		}
		else if (tbl.class == "LicTable" && cur != null) {
			var lines = tbl.tr.td[0].p.content.split('\n');
			cur.address1 = $.trim(lines[0]);
			cur.address2 = $.trim(lines[1]);
			cur.phone = $.trim(lines[2]);
			data.List.push(cur);
			cur = null;
		}
	});
};

Now I’ve got all my data added to a JSON data model on the client side. With this in hand, it became pretty straightforward to:

• Transfer the data to a server-side data model with an ASP.NET Script Service.
• Persist the data to a flat file with XML Serialization.
• Geocode the addresses to latitude/longitude pairs with the Google Maps API.
• Display a pin for each location on a map, along with a detail pane, where clicking on the pin or the left-side summary would highlight the other.
• Add a textarea to the left-side summary so that we can take down notes when we call each daycare location, then save that data server-side as well.

The possibilities for extension are endless, but with this level of sophistication, my wife and I were able to pick out several home day cares that are located conveniently close to the route of our commute, and start with that list when making our calls.

Of course as it turns out, we are grossly ahead of schedule, and most calls resulted in being told we were calling way too early.

But as a software developer, I’ll never object to being called ahead of schedule.