On Friday, around 4 p.m., 122 students who had applied for binding early admission to Vassar saw what the school later called a “test letter” congratulating them on their acceptance. Hours later, the students received a message saying the letter had been posted in error. Once the correct decisions were displayed, only 46 of the students were told they had been accepted.

Vassar has since sent apologies to the students, to all of us alumnae/i, and other interested parties.  They are refunding the application fees for the rejected students. There has been some debate in the blogosphere and elsewhere about whether or not these applicants should have been admitted anyway.

For technologists, this speaks to the cost of the error, but the interesting part is the underlying cause.  This looks to me like yet another example of the bad things that happen when testing is done in the production environment. In this case 76 teenagers involved in what was already a very stressful process were forced to ride an emotional roller coaster of disappointment and embarrassment.  In other cases there can be large financial and even regulatory consequences when test transactions were accidentally put onto production systems and sent to counter-parties. Most senior technologists know a horror story or three on this topic, although they don’t like sharing them.

While there is increasing recognition of the need to better segregate production and test environments, it is both expensive and inconvenient to do so. In times of budgetary restraint, projects to fix this problem can get postponed until the next accident happens. I think companies would be wiser to work on this gradually and incrementally.  Trying to do nothing and trying to do everything all at once are equally unrealistic.

This explanation drew snarky hoots of derision from The Register and Gizmodo, both of whom ridicule the notion that a piece of free software could cost anything to manage.

Clearly, you don’t have to actually know anything about managing IT to write about it for these publications.  There is no such thing a “free” software, if by that you mean that the total cost of ownership is zero. Here’s what it takes to deploy Firefox to tens of thousands of desktops:

• Decide what lockdowns you need in your environment and build a local build of Firefox that implements.
• If you care about plugins, include in the lockdowns a restriction that plugins come from a local repository of approved ones.
• Package it.
• Distribute it.
• Support it.
• Rinse and repeat for each patch release.

What you can’t do in an environment where the user desktop is a managed resource is have users download and self-maintain a complex security-sensitive piece of software.  I’ve worked in organizations that decided that the costs of doing the above was worth spending.  But there was no illusion that supporting Firefox was free.  Even a “best effort” support model requires people to execute it.

One encouraging note was that lots of IT professionals gave these articles the comments they deserved.

Then they lost the stick.

With a yellow sticky attached to it with the password.

We really can’t make this stuff up.

Of course, you may be thinking:  The world didn’t come to an end.  Clearly, this whole thing was just a Y2K hypefest.  I’m sorry the bad guys aren’t quite the eschatologists some people would like them to be, but somebody’s been investing extraordinary amounts of resources making a worm very difficult to kill.  It’s not like there was a contingent of rogue coders, sitting around figuring out where they could put two-character date fields after January 1st, 2001.  There’s a bad guy out there, and while we shouldn’t panic, we shouldn’t quite ignore the situation either.

I agree with his advice.  Don’t panic, but don’t drop your guard either.  Now that network based scanners can spot this, the owners of enterprise networks should be scanning and cleaning up.