Troy Hunt

Request Validation, DotNetNuke and design utopia

2010-03-09T07:00:00.001+11:00

It’s a hot summer day in Perth over on the western seaboard of Australia and the local pub is packed with patrons downing cold beers. You’re in your shiny new Ferrari – red, of course – and come cruising past the pub in full view of the enthralled audience. As any red-blooded, testosterone fuelled Aussie bloke would do, you give the Italian thoroughbred a full redline launch to the delight of the crowd. Right up until you run into the street sign:

Why did this happen? Well there’s the fact the guy was allegedly drunk which is never a good idea in any car, let alone a supercar with a few hundred kilowatts. But there’s also an interesting element of the story captured in the news article:

Witnesses said the driver revved the Italian stallion at the lights next to the Windsor Hotel, at the intersection of Mends Street and Mill Point Rd, dropped the clutch and burned off around the corner

Ah, dropped the clutch and burning off says our inebriated show pony hit the button with the car and the squiggly lines^✝:

As many car manufactures now do, Ferrari fitted the 360 with traction control. Left on, which is the default, loss of traction will cut power to the wheels allowing the vehicle (and hopefully the driver) to regain control and continue on without causing a scene. However, turn it off and suddenly the driver has to take a whole lot more responsibility for their actions because the safety net is gone. This is precisely what Microsoft did when they introduced Request Validation to .NET.

✝ Ok, I’m wildly speculating for the sake of edutainment. And yes trainspotters, I referenced “dropping the clutch” then used an image of the F1 gearbox which obviously doesn’t have a manually actuated clutch. Edutainment!

About request validation

Back in the days of classic ASP, pretty much any data could readily be sent to the server through input mechanisms such as form fields and query strings without any parsing beyond what the competent developer would apply. And here’s your first problem; the term “competent developer” is a little like “common sense” in that neither are that common. Many devs simply did not (and still do not) understand what needs to be parsed and what the potential ramifications are when it doesn’t happen.

Why is request validation even important? Put simply, because you want to control the information which gets sent to your application and potentially stored or redisplayed. Failure to do this opens up a whole range of potential exploits to those with malicious intent.

XSS

Before I delve into specific vulnerabilities, I want to quickly draw attention to the Open Web Application Security Project or OWASP for short. OWASP defines a “Top 10” of the common web application security risks and has become the de facto guideline for all web developers conscious of security implications in their apps.

Onto the vulnerabilities; one of the most common exploits that take advantage of missing input validation is Cross Site Scripting or XSS. Here’s what OWASP has to say about XSS:

XSS flaws occur whenever an application takes untrusted data and sends it to a web browser without proper validation and escaping. XSS allows attackers to execute script in the victim’s browser which can hijack user sessions, deface web sites, or redirect the user to malicious sites.

Let’s break this down a little. “Untrusted data” refers to data sent to the application from third parties outside those explicitly trusted by the application, for example form and querystring inputs. “Proper validation and escaping” involves ensuring the data, although untrusted, is consistent with what is expected from an external party.

Worked examples

XSS exposes itself in a number of different forms but the most commonly seen and easily identifiable exploit involves rewriting querystring parameters directly to the page without validating or escaping characters. Here’s a classic example of what immediately appears to be a potentially vulnerable URL:

http://troyhunt.com/Default.aspx?Name=Troy

In this use case, my name would appear somewhere within the page. Changing the querystring value to “John”, would change the name displayed on the page accordingly. This in itself may pose a vulnerability (direct rewriting of page content) but there are cases where this is valid. It’s the next step in the discovery phase which unearths the vulnerability. What if we changed the URL as follows:

http://troyhunt.com/Default.aspx?Name=Tr<script>alert(‘Hello’);</script>oy

There are three likely scenarios:

The page will identify that a script tag is not valid input for the querystring parameter and the application will reject the request.
The page will display the name exactly as it is entered in the querystring by escaping the non alpha numeric characters and rendering them in HTML as Tr%3Cscript%3Ealert(%91Hello%92);%3C/script%3Eoy which will then display on the page precisely as entered in the querystring.
The page will display the name “Troy” and you will get a JavaScript alert saying “Hello”.

This example is seemingly harmless but stop and think for a moment what is going on in the third scenario. You are controlling not only the HTML content which gets rendered to the browser but the script which executes within it. Let’s run through the possibilities for a moment:

You can rewrite the page contents by closing the tag where the text is displaying and entering your own.
You can insert CSS which changes the layout of the page and substitutes it with your own.
You can write tags which look like username and password fields with a submit button which sends the credentials to your server.
You can insert an IFrame which loads malware from your server and prompts the user to install it.
You can access the user’s cookies for the site (which may include sensitive data if poorly designed) and submit them to your own server.
You can do all of the above in external .js or .css files by including external references.

There are some great examples in the Cross Site Scripting FAQ over on the cgisecurity.com site which talks about XSS in more detail and provides some excellent references if you want to see how much further the exploit can be pushed.

What makes these exploits particularly cunning is that as far as the user is concerned, they’re visiting a legitimate site. The domain looks valid and they’re actually seeing what they probably expected to, there’s just other stuff going on in the background.

Social engineering

In the example above, there is a dependency on someone loading a maliciously formed URL. There are other XSS exploits which are more subvert, such as successfully injecting the examples into a database which then renders them to unsuspecting users, but let’s focus on this one for the purposes of this post.

Tricking someone into following an exploited URL is simply a matter of social engineering. This term refers to manipulating the behaviour of an individual so that they perform an action which may not be overtly dangerous. It’s simply trickery on behalf of the party with malicious intent.

Encouraging the loading of a malicious URL could be done by distributing it through popular channels, such as social networking, with the promise of an alluring website. If the unsuspecting party can be convinced the website is trustworthy, often because the domain and website content both appear legitimate, then the battle is halfway over. It’s then a matter of exercising any number of the potential XSS exploits.

.NET request validation

The entire preamble so far was so that the XSS technique and potential exploit was clear. As I said earlier on, in the old days of classic ASP the exploits above could easily be exercised if the developer hadn’t explicitly parsed input. Then along came .NET and with it the concept of Request Validation.

The beauty of Request Validation is that like the Ferrari’s traction control, it’s just there. You don’t have to turn it on, you don’t have to write code, and in fact you don’t even need to know it exists. It just works. To demonstrate this, here’s an entirely random querystring on a legitimate .NET page with custom errors turned off:

In this case there was no vulnerability to potentially exploit but Request Validation rolled out the safety net and identified a potentially malicious request anyway. This is a very good thing because no matter how (in)competent the developer is, this particular XSS mechanism cannot be exploited by many common string manipulation techniques.

However, there is a downside to the catchall approach. There are legitimate purposes for posting HTML tags to the server. One very common example of this is rich text editors. Many of these allow for editing in WYSIWYG or directly in HTML and as such this becomes a use case for legitimately posting HTML formatted strings to the server. However, just like the traction control button, Request Validation can be simply turned off and it can be done at the page level:

<%@ Page Language="C#" ValidateRequest="false" %>

Whilst this leaves that particular page vulnerable if no manual input parsing is performed, the scope is limited and all other pages still benefit from the fall gamut of Request Validation protection. The other alternative is to turn off Request Validation across the entire application which leaves the web.config looking like this:

<configuration>
  <system.web>
    <pages validateRequest="false" />
  </system.web>
</configuration>

To get a sense of how easy it is to slip up when no global input validation exists, take a look at A XSS Vulnerability in Almost Every PHP Form 'I’ve Ever Written. Or XSS Woes for a similar experience. Very innocuous development practices, very big XSS holes.

The DotNetNuke story

For those not already familiar with this product, DotNetNuke is an open source, .NET based content management system that has now been around in different forms for about 8 years. It’s achieved a strong following by those wanting to quickly and easily deploy CMS and portal style functionality in the .NET realm without the financial and administrative burden of SharePoint.

Being a content management system, there are a lot of places where text, rich text at that, needs to be submitted to the server. A decision appears to have been made to completely disable Request Validation at the DotNetNuke application level as demonstrated in the web.config file above. What this means is that the native protection described above is gone – the traction control is permanently off - and if the developer adds new features and hasn’t explicitly coded against the potential for an exploit, they could be vulnerable.

Unfortunately for DotNetNuke, in November 2009 they identified a vulnerability with their own codebase which spans versions 4.8 through 5.1.4 or to put it another way, nearly two years worth of revisions. This is a significant period of time and as such, a significant number of web applications exist out there with this vulnerability. In fact I found many vulnerable sites after just a few minutes of running carefully crafted Google searches.

The bottom line is that many DotNetNuke sites now sit out in the publicly facing domain with a very easily compromised exploit. This happens in part because the input on the search page was not correctly parsing input but this alone would not have been a problem if Request Validation hadn’t been turned off.

Design utopia

When I came across a site with the vulnerability described above I fired out a quick tweet and got an interesting response. The feedback from DotNetNuke Corp was that .NET Request Validation was a “crude filter” and that it didn’t make sense in a product allowing online editing of HTML.

What Joe was trying to say (and elaborated on in subsequent tweets), was that Request Validation doesn’t provide the same fidelity as custom parsing of request content. And he’s absolutely right. I got similar feedback from a query on Stack Overflow with the response explaining how global input validation “makes programmers lazy” and that they “must be responsible for the vulnerabilities they create”. And I couldn’t agree more.

The problem with the feedback above is that it’s like saying we should just teach people to drive properly in the first place and forget about the traction control. Yes, we should and it’s an admirable pursuit but it’s simply not going to happen and we’ll need to continue catering for those with ambition that exceeds ability. In a utopian world, all developers would understand the OWASP Top 10 and wouldn’t introduce vulnerabilities into their code. But this just doesn’t happen which is why we have Request Validation along with web server level defences such as IIS UrlScan and browser level defences like the IE8 XSS Filter.

Unfortunately the reality is that people will continue to crash cars they are unequipped to handle and developers will keep writing vulnerable code because they don’t understand fundamental security concepts. Just as our friend in the Ferrari had the traction control button, we have a safety net for developers building on ASP.NET. If you turn either of these off, you’d better really know what you’re doing because you’re now on your own without a safety net.

The no-name infrared IP camera for DIY baby monitoring

2010-02-25T07:46:00.001+11:00

As a new parent, I obsess about what the baby is doing. Is he awake, asleep, sucking his thumb or even still breathing? I mean I want to be quite, just not too quite. Do I try and sneak in commando style just to make sure he’s all good and risk waking a sleeping baby (this is never a good idea!), or do I sit in anticipation waking for the baby monitor to confirm signs of life? I’m sure new parent paranoia is not unique to me but I like to have a little more control over my environment than just wondering what on earth is going on behind that closed door.

Recently I placed a laptop in his room with a webcam then fired up a Skype session and monitored him from the desktop in another room. I actually found it disturbingly addictive watching the actions of another whilst obscured from vision however I assure you my voyeurism begins and ends with him! The potentially creepy aspect of this aside, it was interesting to watch him go through sleep cycles and how he behaved as he woke up. It was also pretty darn amusing to see his reaction when my voice came out of nowhere telling him to “GO BACK TO SLEEP” :)

This setup was enlightening but unsustainable and impractical. It also became pretty useless once it got dark but by now I was excited and a more permanent solution was on the cards. Enter the IP camera.

About IP cameras

When it comes to cameras, and by this I mean the video kind, for this sort of thing there are a few different routes you can go down:

Dedicated device and receiver unit: these are expensive as you’re getting a display unit with the package
USB camera: this is essentially my prototype which has a dependency on being plugged in to a PC
IP based camera: this can be wired or wireless and obtains its own IP address making it a network connected device

The advantage of the IP camera is that it is a fully autonomous unit with no dependency on other devices other than a router to give it a network identity. Wireless versions are also ubiquitous which is pretty handy if you don’t have ethernet cabling all over the place. Finally, exposing data over IP opens up a world of possibilities in terms of how it is received; PC, mobile device, internal network or public internet. Lots of options to play with.

Choosing a camera

There are a heap of wireless IP cameras on eBay but the one which keeps coming up is the little guy on the right. I say “little guy” because I have absolutely no idea who makes it or what the model is. Other than being described as “IP Camera”, it basically has no name.

Moving on, as well as being IP based and wifi enabled, this particular device supports night vision, remote pan and tilt, has a built in mic and speaker plus has a bunch of software features including motion detection, image capture, FTP, mail, externally accessible viewing and support for mobile devices. Not bad for under $90 Aus delivered or about $80 American, especially considering the specs:

Image Compression Format	Standard M-JPEG
Sensor	CMOS,300,000 pixel
Image Resolution Rate	VGA(640x480)/QVGA(320x240)
Network interface	RJ-45/10-100 Base T ,802.11b/g
Network protocol	TCP/IP,FTP,SMTP,HTTP,UDP,DHCP,NTP,DDNS,UPNP,D NS,PPPOE
Image Max Transmission Rate	30 frame/second（QVGA）, 15 frame/second（VGA）
Alert control	Output1 router（5VDC，0.1A）；input：1 router（closure Trigger）
Motion Detection	Support
Software Update	Users automatically upgrade
Monitor Mode	IE browse or special program
Playback Mode	Microsoft Media Player
Security	3rd ranks password authority setting
Minimum illumination	2.0Lux@550nm
Auto White Balance	Support
Working environment	-10C°– 50C°，20% - 80%PH

The seller I ended up going with was savemoneyforyou in Hong Kong and I had it on my desk two weeks later which isn’t too bad given the price which obviously included bargain basement shipping. This particular seller was pretty good in answering a couple of questions very promptly and at the time of writing had 99.3% positive feedback from 4,654 buyers. That’s a pretty acceptable risk for 90 bucks.

What you get

You get a box (no name!):

And you get a whole bunch of bits inside including a power supply (with Australian socket converter and a ridiculously short cable), a mounting bracket with an extra base plate and adjustable thumbscrews, a mini CD, an antenna and of course, a camera:

You don’t get a manual, all the info is on CD which is just fine. What’s not so fine is the quality of information in the PDF manual. This is Engrish personified and in some cases it’s almost impossible to understand what’s being said. But hey, who ever reads documentation anyway?! One thing I did enjoy reading though was the mission statement on the final page:

Indeed.

Making it work

On the disk you get a little executable which allows you to find the device by browsing the network. Plug it into wired ethernet and by default, it grabs 192.168.1.126 and exposes the device over port 81.

Pointing IE to the IP gives you an ActiveX installer which then fires you into the video and voila, webcam running!

Other than the obvious video image, you also get controls to pan and tilt plus set the thing “on patrol” which means it spins around all over the place. Major geek-out factor when you first see it moving! Below the controls for resolution, refresh rate, brightness and contrast you also get the ability to record in AVI, snapshot in JPG, listen to audio or broadcast your own through the built-in speaker. The final cog icon at the bottom launches you into the configuration.

Configuration

You’ll find most of the usual IP appliance options here; device info, account configuration, wireless settings etc. You also get all the options to distribute output via means such as mail and FTP or trigger an alarm based on an event.

The only thing I really needed to configure was my time zone and wifi settings which I obviously did whilst still connected directly to the router by ethernet. However, no matter what I did I couldn’t get the device to connect while MAC address control was enabled or while the SSID was being broadcast. I’m not real happy about this and will persevere some more but at least the WPA configuration went smoothly.

Bundled software

Quite frankly, the less said about this the better; it’s a disaster zone. Other than the little IP Camera Finder app mentioned earlier on, everything else is just a train wreck of bad usability, total Windows app design inconsistency and functional flaws. Let me demonstrate; the following is a screen grab of the “IP Camera Super-Client”:

I have no idea what this does as it won’t load without throwing an exception. Maybe it’s not compatible with Win 7 x64 but I have no way of knowing because there’s no app specific documentation. Instead, you get sentences like this (take a deep breath before reading):

When finishing adding camera, once needs to do other settings, please select the camera, and right click to choose equipment setting and then set the following parameters, also when click one monitor screen, click , and set the camera, basic information, equipment parameters, alarm, record, action plans, additional information will appear;

And then there is no additional information. When you try to exit out of the app and are prompted with “Be sure to close the procedure?” after which you’ll get another exception and you get to go through the whole process again. Resource Manager –> End Process.

But wait, there’s more; next up is the impressively named IP Camera Central Management System. It’s a bit hard for me to say exactly what this does because no matter what I did it couldn’t find the device. From the information in the manual, it appears to facilitate scheduling of events and auto recording of activity. What really worried me though was a certain odd behaviour when I was trying to get the thing to run. Look very carefully for anything unusual in the following image:

Yep, it’s a big freakin’ Mickey Mouse. My first reaction was “oh crap, I’ve just installed some Chinese software and now I have a Disney invader” but his behaviour seems pretty innocuous beyond this app. He only appears when running dual screens and moving focus off the app then he disappears a couple of seconds after the mouse stops. There’s no unusual network activity on Resource Monitor while this is going on and everything else on the machine seems to be business as usual so hopefully it’s simply either the world’s most poorly hidden Easter egg or professionalism and usability just mean something different where this thing was built. The final straw with this app is that you need to enter your password to exit it. C’mon, I’m trying to stop doing things do I really need to authenticate?!

In short, don’t install the software. The browser based edition works just fine, at least for my purposes.

Image quality

There’s some interesting behaviour here so let me start with a control photo. The image below is my home office as seen through a Canon DSLR:

And here’s the IP cam in the same light:

Obviously you’re not going to get anywhere near the quality of a DSLR and the overexposure is no surprise but what I did find odd was the behaviour of the blacks. The throw rug and cushion on the couch plus the camera bag to the left of the chair all came out a light blue. Intriguing though, the chair itself shows the full black appearance it naturally has. Even more intriguing was that as I sat in the chair, in my black shirt, the image I saw on the camera showed me in light blue. Here’s the same DSLR versus IP cam comparison:

Picture perfect image reproduction was never really important but this behaviour did take me by surprise. I can only assume it’s somehow related to the filters or infrared capabilities of the device. Which is a good segue to the next image. Here’s how the same shot looks with no light. And by “no light” I mean during the night with no sunlight creeping through the shutters and every single light turned off:

This image is real pleasant surprise. I honestly didn’t expect a $90 “night vision” webcam to have anything like the clarity in the above shot. This is more than sufficient for the purposes of “baby spying” and clearly exposes every feature in the room. The only letdown is the red dot in the middle of the chair which is obviously a defect. Similarly, you can see what looks like a tiny white lens flare in the centre of the black part of the chair in the previous image. Bit of a shame but again, it’s hard to expect perfection from 90 bucks.

You start to get a bit of a gist of how enough light to create the image above is generated when you see this thing running in the dark from the front. A total of 10 red LEDs surround the lens once light levels drop sufficiently (the green LED indicates activity and is also visible under fully lit circumstances):

Having a bit of a trawl through the web, it seems that higher quality IR cameras have less visible light emitting from the diodes and that the appearance above is pretty typical of what the low entry price gets you. Higher quality LEDs apparently emit light at longer wavelengths whilst the cheaper versions, such as the one above, have shorter wavelengths which creates the visible reddish or pinkish glow.

iPhone compatibility

This was important to me as both my wife and I have iPhones and we needed them to work while wandering around the house. The web interface comes with three different mechanisms for viewing the video feed:

“ActiveX Mode” for use in IE which obviously uses an embedded control
“Server Push Mode” for other browsers which pushes down a stream of JPG images
“Mobile Phone” which just refreshes a JPG every few seconds

The mobile phone mode runs pretty well (see image to the right) but the controls are very clunky to use. I found Safari on the iPhone does a pretty good job of rendering the “Server Push Mode” anyway which pretty much makes the mobile version obsolete for my purposes.

Server push mode renders with all the controls surrounding it as seen in one of the first images above. Neither the “picture frame” effect nor the controls are really needed for the purposes of mobile viewing so I’ve just bookmarked the video stream itself directly which is at /videostream.cgi on the device.

Audio

Just don’t even bother. As soon as either the mic or the speaker is activated the frame rate drops dramatically. I suspect this is more to do with the processing power of the device than the increased bandwidth but either way, the end result is the video becomes pretty useless. It doesn’t really concern me as we have an audio baby monitor but it could be frustrating in other circumstances.

Let the baby spying begin!

So configuration complete I installed it just above the little guy’s bed at an angle that gave a pretty full picture of what’s going on. The camera simply remains turned on then once he gets put to bed we get a nice 640x480, view of the following:

This is just what I was after! The software sucks, the audio is dismal and I have no idea what the manual says, but to get a picture like the above in total darkness and transmitted wirelessly so that it’s consumable on PC or mobile and to do it all for only $90 is a very pleasing result.

Creating Subversion pre-commit hooks in .NET

2010-02-13T20:49:00.002+11:00

A while back I wrote about Creating your own custom Subversion management layer which involved rolling your own UI in .NET to perform common management tasks in SVN such as provisioning a repository or managing permissions. This is a great way of quickly and easily giving users a self-service mechanism for managing their own repositories in a controlled, secure fashion.

Continuing the theme of customising SVN to do your bidding I thought I’d share some info on commit hooks. There are a heap of examples out there in Python and Perl but not much in the .NET realm so hopefully this will make someone’s life a little easier.

As with the previous blog post, all the info in this post relates to a Visual SVN Server instance of Subversion. Having said that, there’s nothing specific to this particular SVN distribution so the concepts and code snippets should be equally relevant to any others.

About commit hooks

A common SVN practice, and a good use case for writing a bit of code, is to create event hooks which fire at certain points in the transaction lifecycle of a commit to a repository. Valid transaction lifecycle events include start-commit (before a transaction is created), pre-commit (the transaction is complete but not committed) and post-commit (the transaction is committed and a new revision has been created). There are half a dozen other hook events relating to different server events but we’ll ignore them for the purpose of this post.

Let’s look a bit more into what the SVN book has to say about the pre-commit hook:

This is run when the transaction is complete, but before it is committed. Typically, this hook is used to protect against commits that are disallowed due to content or location (for example, your site might require that all commits to a certain branch include a ticket number from the bug tracker, or that the incoming log message is non-empty). The repository passes two arguments to this program: the path to the repository, and the name of the transaction being committed. If the program returns a non-zero exit value, the commit is aborted and the transaction is removed. If the hook program writes data to stderr, it will be marshalled back to the client.

By far the most common use case for pre-commit hooks is ensuring a revision is accompanied by an appropriate comment. “Appropriate” may simply mean characters must exist or it could lay out some rules for what constitutes an acceptable pattern. Another common use case is to prohibit certain file patterns. For example, the Thumbs.db file really shouldn’t exist in your repository and you may want to enforce its exclusion.

How a commit hook is invoked

As with most things Subversion, it’s pretty simple. If a correctly named executable or script exists in the “hooks” folder of the repository it will be executed at the appropriate time in the transaction. Here you can see a series of template files automatically created by SVN when the repository is provisioned:

Each template file has a sample Unix script inside so if you’re running in that environment you can just drop the .tmpl extension, give it execute rights and you’re good to go. I’m going to write .NET based hooks in this example but just for a sense of what comes out of the box, here’s the contents of the pre-commit.tmpl file:

REPOS="$1"
TXN="$2"

# Make sure that the log message contains some text.
SVNLOOK=/usr/local/bin/svnlook
$SVNLOOK log -t "$TXN" "$REPOS" | \
   grep "[a-zA-Z0-9]" > /dev/null || exit 1

# Check that the author of this commit has the rights to perform
# the commit on the files and directories being modified.
commit-access-control.pl "$REPOS" "$TXN" commit-access-control.cfg || exit 1

# All checks passed, so allow the commit.
exit 0

There are two main behaviours this script has in common with the one I’ll write in this post:

There are two arguments being passed to the script declared as REPOS and TXN.
There is a response code returned being a 1 for a failure or 0 for success.

Creating the solution and reading the args

As mentioned above, all we need is an executable so let’s just build a console app called “SvnPreCommitHooks”:

You’ll get a Program.cs file to act as the entry point to the console app. Let’s start off by following the pattern in the template file above and declaring variables for the two arguments that will be passed to the hook:

namespace SvnPreCommitHooks
{
  class Program
  {
    static void Main(string[] args)
    {
      var repos = args[0];
      var txn = args[1];

The repos argument is the full path of the repository which going by the folder structure in the grab above will be c:\Repositories\BlogTemplate. The txn argument is the commit transaction name and this is something we need to understand a little bit more about before we proceed.

Understanding transactions

The commit transaction name is generated by SVN and comprises of the next revision number and the transaction number in a format such as “3-6” (revision 3, transaction 6). If the transaction succeeds and becomes a new revision then the next txn value will be “4-7”. If it fails then no revision will be created and the next txn value will be “3-7”. The Subversion book explains the process in Understanding Transactions and Revisions:

Every revision begins life as a transaction tree. When doing a commit, a client builds a Subversion transaction that mirrors their local changes (plus any additional changes that might have been made to the repository since the beginning of the client's commit process), and then instructs the repository to store that tree as the next snapshot in the sequence. If the commit succeeds, the transaction is effectively promoted into a new revision tree, and is assigned a new revision number. If the commit fails for some reason, the transaction is destroyed and the client is informed of the failure.

You really don’t need to understand the inner workings of SVN transactions to build hooks but one important point to comprehend is that in order to reject a commit at the pre-commit event we still need to create an entire transaction on the server. This means transferring all the intended commit content to the server which, depending on the data being committed, may mean waiting while large volumes of information is being transferred. And your commit could be rejected causing you to remedy the root cause and go through the entire process again!

Using svnlook to inspect the transaction

So the pre-commit arguments tell us where the repository is and what transaction we need to inspect before promoting it as a new revision. The next thing we need to do is inspect the transaction so we can understand exactly what it is the author is trying to put into the repository. This is where svnlook comes in.

The svnlook command comes packaged with Visual SVN Server and the location is added to the “path” system environment variable on install so it can be invoked from any directory on the system. To get an idea of the sort of information this command can retrieve, here’s what it reports for the last log message and then the last changed paths in the BlogTemplate repository:

The log message is pretty straight forward but let’s look further at the changed paths. What we’ve get here is a collection of rows with each one specifying an action (“D” for deleted, “U” for updated and “A” for added) followed by the path of the file within the repository. In the example above I’ve ditched the .suo file (solution user options shouldn’t be maintained under source control as they’re user specific) and updated the “Blogger Template.xml” file. The commands above have only specified a repository path argument for the respective subcommands but another valid argument is the transaction which is what we’re going to look at next.

Back to the console app; we’ve got the repos and txn arguments and we now know what svnlook can do with them so let’s tie it all together:

private static string GetSvnLookOutput(string repos, string txn, string subcommand)
{
  var processStartInfo = new ProcessStartInfo
  {
    FileName = "svnlook.exe",
    UseShellExecute = false,
    CreateNoWindow = true,
    RedirectStandardOutput = true,
    RedirectStandardError = true,
    Arguments = String.Format("{0} -t \"{1}\" \"{2}\"", subcommand, txn, repos)
  };

  var process = Process.Start(processStartInfo);
  var output = process.StandardOutput.ReadToEnd();
  process.WaitForExit();
  return output;
}

The subcommand parameter is going to allow us to reuse the method for both “log” to get the commit comment and “changed” to get the files and actions. The ProcessStartInfo class allows us to reference both the svnlook command and the parameters after which the Process class will run svnlook and give us back the output in a string. We’ll go back up the entry point of the console app and save the output to variables:

var log = GetSvnLookOutput(repos, txn, "log");
var changedPaths = GetSvnLookOutput(repos, txn, "changed");

Validating the log message

We’re going to say a valid log message must comprise of at least 20 characters and 5 words so let’s encapsulate that within another method. As the message could be invalidated by two different conditions we’re going to give the user a bit of information about what went wrong rather than just returning a boolean. No errors will mean a null message:

private static string GetLogMessageErrors(string log)
{
  if(log.Length < 20)
  {
    return "Message is less than 20 characters.";
  }
  if(log.Split(' ').Length < 5)
  {
    return "Message is less than 5 words.";
  }
  return null;
}

Validating the changed paths

Moving on to the changed paths, cast your mind back to the svnlook output further up. We’ll start by removing the trailing line return with a TrimEnd then split the changedPaths string lines based on Environment.NewLine. Each row then consists of a character to represent the change type, 3 spaces than the changed path which obviously contains the file name. Let’s extract all these out into variables then apply a little conditional logic:

private static string GetFileNameErrors(string changedPaths)
{
  var changeRows = Regex.Split(changedPaths.TrimEnd(), Environment.NewLine);
  foreach (var changeRow in changeRows)
  {
    var changeType = changeRow[0];
    var filePath = changeRow.Substring(4, changeRow.Length - 4);
    var fileName = Path.GetFileName(filePath);
    if(changeType != 'D' && fileName == "Thumbs.db")
    {
      return "Thumbs.db file was found.";
    }
  }
  return null;
}

One thing worth noting here is that we’re only going to return an error if the change type is “D” for “Delete”. Although this condition should be impossible if the hooks are applied to a brand new repository (the file could never have been added in the first place), we may well apply the pre-commit hook to an existing repository. In this scenario the hook would still prohibit adding or changing the Thumbs.db file but obviously we’d like to retain the capability to remove it.

Exiting the hook

That’s pretty much all the legwork done, we now just need to invoke the two methods created above and exit the program with a message and either a success or failure:

var logValidation = GetLogMessageErrors(log);
if(logValidation != null)
{
  Console.Error.WriteLine(logValidation);
  Environment.Exit(1);
}

var changedPathsValidation = GetFileNameErrors(changedPaths);
if (changedPathsValidation != null)
{
  Console.Error.WriteLine(changedPathsValidation);
  Environment.Exit(1);
}

Environment.Exit(0);

It’s pretty obvious from the above but an exit code of 1 will tell SVN to rollback the transaction while 0 indicates success. In both the error states I’ve included a bit of information about what went wrong to try and help the user rectify things on the next go.

Joining all the dots

All we need to do now is compile the project and get the executable invoked on pre-commit. To save on redundancy, we’re going to place the compiled SvnPreCommitHooks.exe directly into the c:\Repositories folder then create a little bootsrapper to place in each repository hooks folder. Let’s save the following as pre-commit.cmd:

C:\Repositories\SvnPreCommitHooks.exe %1 %2

The command file can now be replicated across as many repositories as required without increasing maintenance burden should the console app need to change. It’s also very easy to tie the creation of this file into the custom repository provisioning process I referenced right at the start of this post so that all new repositories can take advantage of the hook.

Testing the hook

This is the fun part! Let’s run through all the test cases using TortoiseSVN:

No message with one valid file and one invalid file:

Fail with an error notification about the message being too short:

Less than 5 words with one valid file and one invalid file:

Fail with an error notification about an insufficient number of words:

Valid message with one valid file and one invalid file:

Fail with an error notification about a disallowed file:

Valid message with one valid file:

Success!

Wrapup

This has been an intentionally simple illustration more to point the .NET reader in the right direction rather than illustrate the potential of SVN hooks. A more applicable real world solution might involve a data driven set of rules or greater integration with other systems such as bug trackers or change logs. There might also be author or even time of day based rules depending on the requirement.

Hopefully this is enough to cover the SVN idiosyncrasies and from here on in it’s all business as usual .NET code wise. SVN is a fantastically versatile product and a few little tweaks like this can really increase both the value of the tool and make for a more productive source control experience.

The hidden costs of building on enterprise software platforms

2010-02-10T07:47:00.002+11:00

Software development has come a long way over the last few decades. We’ve gone from extremely laborious, protracted exercises to create even basic functionality (punch cards anyone?), to the drag and drop, WYSIWYG environment of today. We’ve also gone from a very small number of enthusiast programmers to literally millions of individuals writing software worldwide (there were over 1.3 million software engineers in the US alone in 2008). And we’re all looking for ways to make building software even easier.

As the software evolution has continued, common patterns have been identified and dedicated tools have emerged to abstract these patterns and enable developers to build applications more efficiently. Over time, enterprise software platforms have gained popularity, abstracting common patterns with the promise of readymade frameworks that promote rapid development, greater agility and lower support costs.

But do enterprise software platforms always achieve this objective? Are the gains of rapid development offset by other costs – hidden costs – which don’t expose themselves at the outset? Are these platforms really the panacea they’re cracked up to be?

What’s an enterprise software platform?

Enterprise software platforms attempt to solve abstracted requirements by providing a generic platform which can be applied to solve specific business problems. The examples I’ll refer to most frequently are Microsoft SharePoint and K2 workflow but the principals discussed are equally relevant to products such as Oracle Portal or Microsoft CRM.

Let’s start by understanding what K2 is based on their description:

The K2^® software platform increases business efficiency and makes work simple. Its visual, familiar tools allow people with varying backgrounds and technical skill to create applications that automate business processes and streamline operations. And when something in the business changes, modifying the applications to keep up is easy.

And a little about SharePoint from the Microsoft perspective:

Microsoft Office SharePoint Server 2007 is an integrated suite of server capabilities that can help improve organizational effectiveness by providing comprehensive content management and enterprise search, accelerating shared business processes, and facilitating information-sharing across boundaries for better business insight. Additionally, this collaboration and content management server provides IT professionals and developers with the platform and tools they need for server administration, application extensibility, and interoperability.

Both of these are very vague and fraught with business buzzwords that don’t tell you an awful lot about what the product does. I ran Bullfighter over the SharePoint description just for interest sake:

Bullfighter is a bit tongue in cheek but it has a point about obfuscation and preponderance. Let me try and distil the nature of these products into a more consumer friendly fashion:

As application platforms, these products aim to streamline development by abstracting common solution behaviours and packaging them into products which may be consumed by business applications. They both provide APIs and integrate with IDEs to be utilised at design time as well as providing server based installations which are required at runtime.

One quick clarification about SharePoint as it’s a bit of a multifaceted product; I’m referring to using it as a platform on which to write custom application code as opposed to using the native teamsite and portal collaboration features. These features are fantastic straight out of the box and are in a very different category to using SharePoint as a foundation for writing code.

What’s wrong with this?

Where the problems first arise is in customer expectation, and is it any wonder after reading the quotes from the manufacturers above? Because the positioning of the products is so generic it’s easy for customers to form opinions and set expectations with little understanding of the actual effort involved. The bottom line is that many businesses experience unexpected surprises during the development and support cycles of applications built on these platforms.

To be fair, both these products and enterprise software platforms in general can be fantastic when used in the right way. The intention of this post is to highlight areas which can be problematic in order to contribute to a more pragmatic analysis of the platforms before commitments are made.

The people problem

A serious problem with creating a dependency on these platforms is that you create a dependency on people with niche skills. This severely restricts the potential candidates you can hire either directly or indirectly through an ISV. Remember also that we’re talking about server platforms so you not only need good developers, you need good infrastructure people who understand the product.

Let me demonstrate; Stack Overflow Careers allows developers to promote their CVs online and there are currently a couple of thousand people putting themselves out there. Let’s imagine we’re recruiting for a .NET developer:

Let’s compare that to a SharePoint search:

And now K2:

So for every potential candidate you could hire to develop SharePoint applications there are eight .NET developers lining up. K2 doesn’t even feature. But is this a realistic representation of people availability? Let’s reverse the process and see how many jobs are available on seek.com.au in Sydney for the same technologies:

A little less damning than the Stack Overflow results but you’re still looking at five and a half .NET jobs available for every SharePoint one. The K2 results are a bit misleading because five of them refer to the K2 search engine. The remaining result lists K2 as “beneficial”.

The relatively small representation of SharePoint and particularly K2 poses a number of problems. Firstly, the reduced number of candidates means recruitment is more protracted as good people are harder to find. This poses a direct risk to agility in terms of being able to rapidly scale resources up and down and in turn impacts negatively on those charged with the recruitment process.

Secondly, market forces tend to financially favour those with skills which are not commonly found but are in demand. In turn, these people tend to command higher rates. The last couple of times I recruited people for a SharePoint developer role they were consistently priced around 20% higher than their .NET counterparts. Great for the candidate, not great for the employer.

Finally, it doesn’t matter whether you’re engaging individuals directly or using the services of an ISV, the same risks and costs exist and ultimately they’re passed on to the customer. The same market forces drive the cost and availability of ISVs; there are far fewer competent vendors out there developing SharePoint or K2 than there are .NET and this is reflected in their pricing.

From a cost perspective, all this begins to erode the gains made in rapid development. The fact an application may be built 20% faster on one of these platforms is quickly offset (opportunity cost of earlier delivery aside), by 20% higher resource costs.

The development environment problem

Unsurprisingly, the process of building software on these platforms is a little different to your classic development model. The surprise comes when you understand just how different the development process can be.

Let’s look at the SharePoint approach compared to traditional ASP.NET development. In the .NET approach, the developer has Visual Studio on their everyday work machine and pretty much goes from there. In the SharePoint model, the developer needs to work on an actual SharePoint server. You simply cannot code against the object model any other way.

What the SharePoint development model usually entails is running a virtualisation product such as Microsoft Virtual PC and building a sever environment complete with OS, SQL Server, SharePoint server and finally Visual Studio.

The next thing you need to consider is that running a virtual machine, particularly a server running SQL Server, is very resource intensive. Usually the virtual disk of the machine is placed on an external drive to spare the host of additional HDD activity so there is a small investment required there.

The development machine is also going to need a good allocation of RAM (at least a GB), so this will often mean increasing the physical memory of the host if it’s expected to continuing performing other tasks normally expected of a desktop computer. A machine with 3GB is pretty much an entry point but it will incur a productivity tax. Of course anything beyond much more than 3GB means moving to a 64 bit operating system which adds yet another dependency to the equation.

These are not insurmountable problems but they may come as a surprise to the unprepared. There is a time and expertise dependency in building a virtual machine, a hardware dependency on having a machine capable of running it in an efficient fashion and this potentially even creates an OS dependency because of the 64 bit issue.

A final consideration, relevant to some businesses, is that the development model has a dependency on leveraging an “unmanaged” server installation which may breach corporate desktop rules.

Sidenote: these issues and many more are articulated very well in the Stack Overflow post about What are your biggest complaints about SharePoint?

The shared farm problem

These are expensive, specialised products requiring specialised expertise so it’s only natural that organisations aim to consolidate instances into shared farms. The theory is that by decreasing the infrastructure footprint of these products the business can minimise the costs of licensing, hardware and support. The same rationale is perfectly valid for products such as IIS or SQL Server and at least on the surface, it makes sense to do the same with the enterprise software platforms.

The problem with any shared farm is cohabitation. Any application on the farm is simply a tenant and it has to play nice with all the other tenants so the farm owner creates the server equivalent of strata by-laws. The by-laws define governance designed to ensure the harmony and sustainability of the environment in the best interests of all the tenants. The extent of the governance and the freedom provided within them is largely up to the model implemented by the underlying technology.

Let’s take the example of a website hosting provider offering a typical SQL Server backend and IIS front end on a shared farm. These technologies provide a robust sandboxed model which offers the customer a large degree of both security and autonomy in that in most cases, once the resources are provisioned there is a great deal of freedom in terms of what the customer can deploy to these resources without risking the integrity of other tenants in the same environment. Customers can regularly self-deploy web applications and databases in a secure, on-demand fashion which means in terms of application agility it’s a big thumbs up for the model.

The nature of these products makes the shared farm model a little trickier. It obviously varies between products but let’s take SharePoint as a fairly typical example. One of the shortcomings of this product is that it makes code releases difficult at the best of times (I’m talking about SharePoint 2007 here), let alone when you’re also trying to secure the integrity of other applications in the same environment. Deployment of SharePoint applications packaged as a Windows Solution Package (.wsp) requires permission levels across the environment and beyond the scope of just a simple application meaning the potential ramifications of granting these rights are large. Have a glance through the DevX article about Creating and Deploying SharePoint Solution Files and you’ll get a pretty good idea of how low level a SharePoint deployment is.

The problem this leaves us with is that SharePoint apps can’t just simply be deployed by the developers into a shared farm but rather must be managed by a group that has the best interests of the broader farm in mind. Some people would argue that this model is preferable in any environment, not just SharePoint, but the reality is that many organisations simply don’t implement this degree of governance. The bottom line is that this model of deployment has a level of governance you don’t necessarily find in other technology stacks, such as your classic ASP.NET and SQL Server, and that it can have an adverse affect on agility and cost.

Sidenote: SharePoint 2010 implements Sandbox Solutions which come with the promise of greater flexibility and autonomy to the developer than what we’ve seen in previous versions. However, there are still limitations which will mean the concept of managed deployments will need to persist.

The new version problem

Products like SharePoint and K2 are undoubtedly feature rich and inevitably complex beasts under the covers. They continue to evolve and incrementally improve with each new release and with a product like SharePoint, the release cycle is only a few years. Chances are that any application you build on one of these platforms is going to need to go through an upgrade process at some time.

The problem with both these platforms is that they’re simply not backward compatible, at least not without going through conversion and testing processes requiring various degrees of manual labour. K2, for example, went from heavily promoting the use of Smart Forms as a UI layer yet now recommends standard ASP.NET webforms or InfoPath. There is a K2 BlackPearl Migration Utility available but the feedback on the effectiveness of this and the amount of manual labour required indicates it’s by no means a panacea for upgrading.

In the K2 case you then have idiosyncratic rules such as “all workflows must completely execute in the same version of the workflow in which they were started” which is going to mean some very carefully crafted migrations.

Inevitably businesses take the decision to upgrade these platforms. Nobody wants to go on supporting a single technology version for perpetuity and as new releases become available, support for previous editions wane and the argument for building new applications on the latest product becomes more compelling. Ultimately, the upgrade path needs to be pursued at some point in time and the challenges described above will be faced.

Jumping to conclusions

A risk these products all pose is that they make it very easy for decision makers to jump to conclusions. Building an app with workflow? You’ll be needing K2. Need alerts? That’ll be SharePoint then. They’re reactions based on their understanding of the product as represented by the manufacturer and often communicated at a very “manager speak” level.

These reactions are understandable, to a degree, when you remember that K2 “increases business efficiency and makes work simple” while SharePoint “provides IT professionals and developers with the platform and tools they need”. Of course by now we know it’s not that simple and there are a number of other problems to contend with.

Let me draw on a workflow example to put the problem into context. Let’s say we have a requirement to seek an expense approval based on an ASL. The input to the workflow process is a price and the business logic involves assessing this against a list of potential approvers, each with their own signatory level. There will likely be the ability to view and approve outstanding tasks and email notifications will probably be used to either call people to action or advise them of a status change. This is your ubiquitous Hello World app for workflow.

Should this be a K2 workflow? Quite frankly, no, it shouldn’t. This is programming 101 and most competent developers could put the whole thing together from scratch within a day and without experiencing any of the problems described above. Sure, it won’t provide all the functionality that K2 can but then things like reporting and escalations weren’t part of this particular requirement and are often way down the feature priority list.

The point is that the way these products are positioned makes it easy to prematurely jump to conclusions. It’s easy for this to occur when the true nature of the product is not understood. It’s also easy for this to occur once a business has invested in the project and loss aversion sets in. Neither of these are very good reasons to use an enterprise software platform.

The alternative

It’s simple; custom coding.

The current state of .NET and SQL Server is extremely feature rich, as are many comparable non-Microsoft products. Just within the current generation of the technology we’ve had numerous new features which streamline the development process in a number of ways. Just off the top of my head; LINQ, implicit typing, WCF, WF, FILESTREAM and geospatial data types and the MERGE TSQL function. I’m not saying these entirely take the place of a workflow or collaboration platform but I am saying they streamline the development process and make the implementation of most requirements significantly easier than it was even just a few years ago.

In many cases, as with the one mentioned above for an ASL based approval, custom coding is a more than adequate means of fulfilling the requirement without suffering from the problems already described as systemic of the enterprise framework approach. There are downsides though; some features may take longer to build (indeed they could become quite complex) as you don’t get the same degree of native functionality. But when you can build the entire requirement with custom code in less time than it takes just to configure the development environment for K2, the alternative approach begins to look pretty good.

Edit: Judging by some of the comments I’ve received, this section has possibly been taken out of context and I want to be clear that by no means do I advocate replacing either K2 or SharePoint with custom code in a broad-brush fashion. This example is intended to illustrate that depending on the scenario, there are times where the “roll your own” approach may be more appropriate. Not every workflow requirement needs K2 nor does every collaboration requirement need SharePoint. I’ve been directly involved in projects which have used these tools to great effect and I wouldn’t do them any differently, but I’ve also seen instances here they have been detrimental for many of the reasons mentioned above. Pragmatism and an open minded, objective approach is key; there are no silver bullets.

Proceeding with an enterprise software platform

None of the above is intended to say that these platforms are evil. There is absolutely a time and a place for them but the use case is just not as broad as many people would have you believe. In an environment with sufficient scale to justify the investment, the right expertise to support through the lifecycle and possessing business problems closely aligned to the offerings of these platforms, there may be a valid case to be made.

Let’s take a pragmatic approach to choosing the technology and remain conscious of the following:

Understand the resource needs for building on the platform. They may be harder to find and cost more than you expected.
Plan carefully for the development environment. Ensure the correct hardware exists and that no corporate barriers prohibit working in the recommended fashion. Also ensure greater lead time is made available to ready the environment.
Expect a higher degree of governance if deploying to a managed environment. This may have an adverse impact on agility and come with a financial overhead.
Acknowledge that upgrading the platform as newer versions are released may impose a more significant overhead than you’re used to.

If someone goes through the analysis process and comes out the other side with a K2 or SharePoint based solution, fantastic. They’ve done the due diligence, they understand the risks and they’ve ultimately chosen the best technology for the job.

However, if conclusions are drawn prematurely and an enterprise software platform is committed to without fully understanding the ramifications, you’re going to need a lot of patience and very deep pockets. Caveat emptor!

The commoditisation of the coder

2010-01-29T08:23:00.003+11:00

I love a cold beer. Not just because it’s refreshing and makes me worry less about the world’s problems, but also because of beer’s fungibility. Let me explain; I can go down to the store and buy a beer and it’s pretty much the same as any other beer I might purchase elsewhere. Sure, there are different standards of beer and I’m going to pay a few dollars more for my favourite Little Creatures than I am for VB but in essence I’m still getting hops and yeast with some water.

The point about fungibility is that beer is the same basic product no matter which one it is you’re drinking. In short, beer is a commodity.

Coder commoditisation

Commoditisation occurs once people start believing that a good, in the case of this post a coder, can be acquired without qualitative differentiation. In simple terms it’s the mindset that CoderA == CoderB == CoderC. There are obviously cases where this is true but the commoditisation I’m referring to is the broad-brush assumption that software development is a consistently repeatable process with the same end result from the same amount of effort regardless of the person at the keyboard.

You can spot coder commoditisation in action when you see comments like this:

Don’t worry, we’ll just get unknown CoderB to step in while CoderA is away.

Or the classic financial argument:

Hey, we could get unknown CoderB for 25% less than what CoderA costs!

These are commodity based statements in that they assume CoderA can be mutually substituted by CoderB whilst achieving the same results in the absence of quantifiable knowledge of their skill level.

Building software is simple - if you know where to look

I don’t actually believe that software development is a complex process. Very tricky problems can be solved by smart coders with such ease and grace that the practitioner barely raises a sweat. They know the shortcuts, the pitfalls, the well trodden paths to coding success and when they tie it all together the process becomes simple, elegant and most likely successful.

The key observation here though is that this person is clear about what they’re setting out to do. They have the pieces of the puzzle and a mental image of what it should look like once they put it all together. What’s more, they’ve successfully performed this process many times before. These are the people who take something complex and make it look easy.

Compare this to the coder who simply cannot put the pieces together. Either that or the effort it takes them is substantially different. There are numerous potential causes of this; they may be poor at problem solving, unable to ask the rights questions to understand the requirements, have a poor grasp of the technology or possibly they’re just lazy!

Oils Ain’t oils

Just like engine lubricants, when it comes to coders Oils Ain’t Oils. Castrol would have you believe that not all oils are created equal and the same is true of the coder. When it comes to coding we’re talking about a process which has a huge number of variables in terms of how software is built. The process is then performed by people from extremely diverse backgrounds in terms of culture, education and experience.

One thing that separates software development from many other industries is that people become coders by following very diverse paths. If you want to become a doctor, you go to medical school. If you want to become a lawyer, you go to law school. If you want to become a coder, you start writing software in your basement when you’re 15. Or you start doing it as the “shadow IT” guy alongside the reason you’re actually at your place of work. Or you possibly go to university but often it’s to study engineering. It’s because of this people diversity that we simply cannot expect consistency in either effort or output from the process of writing software.

Quite frankly, some people are just very bad coders. Certainly underperformance is not unique to the software world, you could be a bad chef or a bad cleaner but the difference is that poor quality work is immediately apparent whereas the handiwork of a bad coder often does not become obvious until long after the work is done. Oftentimes the coder has the advantage of relatively anonymous work. I know if my steak is overcooked and I know if the kitchen in the office hasn’t been cleaned but quite frankly I have no idea what happens to my internet banking credentials after I hit the submit button. Unlike the chef or the cleaner, the coder’s work is not usually unmasked as sub-standard unless a peer of equal or greater knowledge is exposed directly to it.

Offshoring

One area ripe for commoditisation is outsourcing to low cost markets. A key rationale for sending work offshore is that resource costs are lower. Why pay $100/h in the Western World when you can send the work to an emerging market at a fraction of the cost? Once again though, this assumes the people can be commoditised based purely on an hourly rate. What this rationale often does not consider is the quality of the resource.

I’m not saying resources in your typical offshoring locations are necessarily inferior. The commoditisation argument would be the same if the tables were turned; without sufficient due diligence to quantify the capabilities of the resources you’re still making the assumption that writing code is a mutually exchangeable activity regardless of the individual.

Obviously the “win-win” situation is to leverage high quality, low cost resources but this only happens when commoditisation is not the sole focus and an understanding the coder’s competence is gained. Only then can the decision be made holistically based on both competence and cost but without both pieces of information only part of the picture is clear.

The Mythical Man Month

So why not worry less about putting high quality coders on a project and just focus on cost? Why not grab twice as many people for possibly a third of the price from a low cost market and not even bother with all the rigmarole of attempting to understand their capabilities? The answer is described in Fred Brooks’ The Mythical Man Month through Brooke’s Law:

Adding manpower to a late software project makes it later

Brooks describes his point through this now popularised saying:

Nine women can't make a baby in one month

In The Mythical Man Month, the author talks about how an increasing number of people in a project increases not only the total amount of project familiarisation which is required (which is linear as numbers increase), but more importantly how much extra communication is required which increases exponentially with greater numbers. Joel Spolsky illustrated this recently in his column about A Little Less Conversation where he demonstrates how connections increase in relation to people. More connections mean exponentially more communication which means less output.

People	Connections
1	0
2	1
3	3
4	6
5	10
6	15
7	21
8	28
9	36
10	45

So by attempting to overcome inadequacies in skill level by doubling the number of coders on a project from say, three up to six, you’re potentially introducing five times as much communication. This level of non-productiveness will very quickly erode cost savings. And you’re still left with an application which although functional, will likely suffer from quality related issues over the longer term.

Commoditisation fallout

Assuming coder commoditisation has occurred and the wrong people are left to run wild at the keyboard, there are a number of ways things can rapidly go downhill:

Required effort to finish tasks becomes high. Seemingly simple jobs become a chore and durations rapidly blow out well beyond what’s required by the competent coder.
Sustain costs increase either due to buggy software or high degrees of effort required for future changes.
Customer satisfaction decreases as durations increase and confidence wanes when expectations are not met.

All of these can be damaging for the project and for the organisation as a whole. It can also be extremely difficult to manage the wrong people out of the roles they’re in.

Coders are not beer

Coders are not very fungible and treating them as interchangeable commodities is simply a recipe for failure. The mindset that units of work can be defined and quantified then assigned to any coder and still get a consistent result is borne of a misunderstanding of what the software development process consists of. Here’s what needs to be understood if we’re to avoid commoditised disappointment:

Coders come from a very broad range of backgrounds and have wildly varying skill levels.
This varying skill level can have a major impact on both the duration of development and the ongoing sustainability of the software.
Merely throwing a greater number of less skilled individuals at a problem will not necessarily solve it faster or more cost effectively.
Software development is a profession requiring skilled practitioners to produce a quality product.

Ignore these at your own peril; I’m off for a beer!

Why ReSharper recommends the “var” keyword in .NET 2.0 projects

2010-01-24T12:10:00.001+11:00

I was a little confused this week as to why ReSharper was recommending using implicitly typed variable declarations in a VS2010 solution targeting .NET 2. Somewhere in my mind I had directly associated the “var” keyword with the release of .NET 3.5 so this looked a little odd to me:

As it turns out, the var keyword is a feature of the compiler, not the .NET CLR. The same is true for automatic properties and object initialisers. The bottom line is that you can use these features in VS08 or VS2010 and the compiler will happily go along with it and translate the code to .NET 2.0 compatible syntax in the object code.

There’s an excellent post on Shahar Gvirtz's blog where he disassembles code using this syntax in Reflector to reveal plain old .NET 2.0 syntax. So in short, implicit typing is fine for anyone running a recent version of Visual Studio and, as usual, ReSharper is correct in identifying this as an opportunity to polish the code.

SVN “Can’t create directory” Error

2010-01-15T17:23:00.001+11:00

Here’s another one of those Subversion idiosyncrasies which threw me the other day and I couldn’t readily find an answer for. When committing a changeset I kept getting the error “Can’t create directory” followed by the the path of the repository on the server then “The system cannot find the path specified”.

The first thing to get clear is that this is a Subversion error, it’s not related to the local working directory nor is it related to Tortoise SVN.

Looking at the path in the image above, you’ll see it specifies the subdirectory “db\transactions”. After inspecting the folder structure of the repository, I found the subdirectory was missing. Comparing it to a newly created test repository I found that not only was the "db\transactions” directory missing but so was “db\txn-protorevs”.

I’m not sure how these folders disappeared. I run a robocopy script to backup my repositories to a NAS device and although the source shouldn’t be touched, the timing is rather coincidental. Whatever the cause, the folders disappeared and this was what caused the issue.

The fix

Really, really basic; just manually recreate the folders. They don’t retain any information post-commit, they just need to exist so transactions can be established. Simple as it may be, I found vey few online references to this error and nothing around the fix so hopefully this will save someone a bit of time in the future.

BTW, small sidenote and a quick plug for some very good software; as you’ll see in the images above, this project is called “TotalBabyReport”. Total Baby is an iPhone app which tracks everything your baby does which is pretty handy when you’re sleep deprived and can’t remember what you had for breakfast let alone when the baby last slept. Or ate something. Only thing is it doesn’t do well is report on trends across time such as sleep patterns so I’ve created a little personal app to try and get some baby business intelligence metrics :)

Foolproof personal backups with Mozy

2010-01-10T17:05:00.003+11:00

It starts with that sinking feeling and all sorts of questions running through your head:

When did I last backup?

Did I include everything?

Does the backup actually work?

When did I last try restoring it?

How much unrecoverable data could I have lost?

Losing data can be an absolute stomach churning experience. Those first moments when the penny drops and you realise you’ve got a big, big problem are absolutely nightmarish and without a robust backup strategy it’s just a matter of time until you experience this firsthand.

This last happened to me a couple of years ago when a disk had a catastrophic failure and refused to boot. Slaving the disk off another machine didn’t do much good so it was off the data recovery specialists and let me tell you, this is not cheap and it’s not fast. I used a group in Sydney called Payam Data Recovery and whilst their service was fantastic, I always felt like each conversation was akin to speaking to your doctor and expecting him to tell you you’ve got a terminal illness and that you need to “start making arrangements”.

Anyway, I got to thinking about the whole backup strategy thing again recently after both Jeff Atwood and Phil Haack suffered data loss on their blogs after a server hardware failure. Being pretty popular guys this lead to a lot of feedback and suggestions from the community as to how to both retrieve their data (some very innovative feedback there!) and how to implement a suitable backup strategy in the future. It brought the whole disaster recovery topic back into focus for me so I thought I’d share my approach which personally, I think works pretty well.

Personal, not enterprise

Let me quickly contextualise this; I’m talking about personal backups, not enterprise data which is probably more consistent with Jeff and Phil’s experience above although many of the principals are the same. Let me typify what I mean by personal data:

Documents such as Word, Excel, PDF.
Audio content such as MP3s (could be from iTunes, ripped from CD, etc).
Photos (predominantly JPG and possibly RAW) and video.
Any other personal content you don’t want to lose, which for me, includes things like my Subversion repositories and database backups for personal projects.

Essentially I’m talking about the sort of content most of us accrue during our daily lives as civilians. It’s the same sort of data your parents or kids or cousins accrue regardless of whether they’re techie people or not.

Manual processes will fail; it’s just a question of when

Here’s a pretty typical backup strategy:

Grab the thumb drive / USB hard drive / blank DVD from the cupboard / safe / relative’s house.
Copy and paste the important stuff onto the device.
Return said device to its storage location.

The biggest problem here is that it requires you to be proactive. You have to consciously think “Wow, haven’t done a backup for a while, might be a good idea” and the reality is this never happens as frequently as it needs to and you carry a lot of risk between backups. What’s more, the process will become increasingly lengthy as the data expands. I could already fill nearly 20 DVDs with my data and it would take a long time to copy the data over via USB to an external device. Inevitably, the process deteriorates and becomes suboptimal increasing the risk of losing important data.

Onsite backups are vulnerable

One thing you need to get your head around and I don’t think most people do, is that if your backup strategy doesn’t involve getting your data outside of your house then you need consider what happens in the event of your backups disappearing. This usually happens one of two ways:

You get burgled and the thieves take anything that has a flashing light or looks valuable which might include your NAS device, your backup server or your carefully hidden thumb drive.
You literally experience a disaster such as flood or fire. 11 months ago we had absolutely catastrophic fires in Australia with the loss of 173 lives and, I suspect, huge amounts of now unrecoverable data.

The point is that if your data and your backups are anywhere near each other then they are both at risk of the same events causing you to lose it. Irrecoverably.

“Last version” backups can still lead to data loss

A typical use case for restoring from backup is corrupted data. This could occur via program error or malicious activity but the net result is still the same; you need to restore from an uncompromised copy. The problem is, if you’re using one of the typical approaches mentioned above and you’re actually using it frequently, there’s a good chance your corrupted copy has overwritten a previous good version. The bottom line is that without the ability to restore from point in time you are vulnerable if files become corrupted. You could take the approach of continually making new copies, rather than overwriting existing ones, but the feasibility of doing this decreases dramatically with increasing content size and frequency of backups.

You need to be able to get the data back again

In the wake of Jeff and Phil’s data loss I mentioned earlier, Joel Spolsky posted about Let’s stop talking about “backups”. His point was simply that although backups are great, you need to be able to reliably get the data back again. This means you need confidence in the backup process, the integrity of the media it’s stored on, the ability to read the content again at a later date and of course that you’ve actually captured everything you need to restore from square one i.e. new machine with zero data. Entrusting media such as DVDs and thumb drives to reliably hold data without physical degradation for possibly many years is again, a risky proposition.

Mozy backup

As far as I’m concerned, you’re really only left with one viable choice if your data is important and that’s using an online service such as Mozy. After my own data loss I spent a great deal of time researching suitable backup providers and although there were a few out there at the time, Mozy is the one that really stood out to me. The status quo may have changed by now but certainly Mozy is still doing a great job of managing my backups.

The purpose of Mozy is pretty straight forward:

Mozy is a simple and safe way to back up all the important stuff on your computer. A copy of your data is stored in a secure, remote location for safekeeping, so that in the event of disaster your data is still retrievable.

It sounds easy, and it is. Mozy provides a small client which runs on your machine and allows you to specify backup sets. By default, you get a bunch of common sets such as photos and documents.

Alternatively, you can just backup based on file system path as I’ve done (more on that later).

Finally, you decide on what sort of schedule you’d like your backups to happen on. By default it will try and do it when the PC is not in use but you can manually define a particular time of day if it suits. I found there was no noticeable performance impact on the PC when it ran (you can actually adjust the balance between backup speed and PC performance if you like) so I just ran with the defaults.

That’s it, you’re done! Any content you place in the specified backup sets or file paths will automatically be backed up offsite.

Restoring

Mozy provides a few different options here, the easiest of which is that your entire backup set gets mapped as another drive you can simply navigate to in Windows Explorer. You see a list of each drive you’ve previously backed up to from which you can drill down to the appropriate path, locate the content you’re looking for then just right click and “Restore”.

That’s it; restoring lost content could not possibly be easier! Obviously there is a bandwidth consideration depending on the volume of content you’re restoring but I’ll touch on that again in a moment.

But what if the latest backup was a corrupted version? Not a problem, right click on any folder, choose “Change time” and you can restore from any point in time in the past so you can easily roll back to a known good version. You can access the same functionality by right clicking on the source file and choosing “Restore Previous Version” which then lists all the previous versions you’ve successfully backed up. Although not its primary purpose, this is also a neat little version control feature.

The other online mechanism of restoring files is via a web interface. This is pretty handy if, for whatever reason, you don’t have the Mozy client on a machine and still want to access backups.

All this works great for restoring a reasonable size collection of files, but what if you’re talking about catastrophic loss and the need to restore tens of GB? Mozy provides a sneaker-ware solution involving burning to DVD and FedExing the content to you. Obviously they’re going to charge for this service but its US$30 plus US$0.50 per GB so even if you’re talking 100GB you’re only looking at 80 bucks to get all your data back. That’s a very small fraction of what I paid Payam to recover my data and even then, some of it wasn’t recoverable.

Security

When I talk to people about this backup strategy one of the most common questions is “is it secure?” It’s a bit of a tough one because there’s no simple yes or no answer. Is anything on the net “secure”? Banks get compromised! Mozy reports using 128 bit SSL for the transport layer and 448 bit Blowfish encryption in storage so at least on the surface of it, the encryption appears to be solid.

Just a couple of other thoughts on security though. Firstly, how secure is your data at present? I mean what is the possibility of someone walking out your PC or your backup device under their arm and reading the data? There might be a password on the PC but I’d hasten a guess that’s about the extent of it.

The other consideration is the balance of impact of loss versus impact of disclosure. Unless you’ve got your own Paris Hilton style home video on your machine, I’d personally be far more upset about losing my data altogether than someone getting hold of my photos or even my financial statements. Passwords can be changed; family photos can never be recreated.

Bandwidth

The other obvious question is “What’s it going to do to my bandwidth?” Obviously large backups are going to chew up a lot of upstream capacity. I just came from holidays with over 10GB of photos and video and it took several days for Mozy to work through it all. In terms of the impact of using this bandwidth, I haven’t noticed any adverse behaviour primarily because I have Mozy configured not to run while I’m using it which means a lot of the network utilisation is happening during the night.

The other big factor is how your ISP handles upstream data billing. Providers such as Telstra BigPond count it along with your downstream traffic but someone like TPG, who I use, don’t count any sent traffic. Even if they did, when you’re looking at less than A$70 a month for 160GB of traffic (ok, half of that is off peak), bandwidth is starting to get pretty cheap.

Cost

First, the good news; you can use up to 2GB for free which means you can download the client, test it out, make sure you’re happy with it then make a financial commitment. After that you’re looking at US$4.95 a month for unlimited data which on my cappuccino-meter (i.e. the price of a coffee which you wouldn’t think twice about spending – around A$4), it’s less than a cappuccino and a half a month which I reckon is very, very good value.

Limitations

There’s only gotcha which really causes me any grief and that is the “Home” version (as opposed to the “Business” version which is significantly more expensive) must backup from a local drive. I use a DLINK DNS-323 NAS device to store absolutely everything because it’s easily accessible from any machine, has built in RAID 1 with two mirrored drives and acts as an FTP server. Unfortunately because this drive is considered external there was absolutely no way I could get Mozy to backup from it.

What I’ve ended up doing is installing a spare disk in my primary PC then configuring Robocopy to mirror all the content I actually want on the NAS to this drive. It runs nightly and in some ways makes things a little simpler as I just tell Mozy to backup absolutely everything on the F drive which is the spare disk. It would be nice to be able to avoid this but I can see where they’re coming from in terms of segregating their business and personal offerings.

Alternatives

There are a couple of alternatives to this sort of approach. Apple has the Time Machine which I hear good things about and Microsoft has Windows Home Server which both provide personal backup solutions. I’m not sure how the respective manufacturers recommend taking backups offsite but this to me is an essential requirement if the backups are to be seriously foolproof. In my case, neither solution was suitable (I don’t have a Mac and don’t particularly want to run a server at home), but they might be viable alternatives for some assuming the data can be safely and reliably stored somewhere redundant.

Summary

In reading back through this post, it does appear overtly favourable towards Mozy but I’m honestly just really, really impressed with the product and am not incentivised in any way to write this. I just love the fact that you install it and forget about it and the only time you’ll ever need to actually think about it is when you need it most. It sure makes me sleep a lot better at night and as so long as I myself don’t suffer a catastrophic failure (incidentally, I have a backup for me as well!), I’m confident I’m never going to be experiencing that sinking feeling again.

Connecting to SQL08 on Windows 7 from a remote machine

2009-12-14T22:39:00.001+11:00

I’ve got both a desktop and a laptop running SQL08 Developer Edition on top of 64 bit Windows 7 which until today, did not play nicely together. I could not get a remote connection from Visual Studio or SQL Management Studio to the other machine nor could I make an ADO.NET connection. Every attempt to connect resulted in a lengthy delay followed by a message such as the following from SQL Management Studio when trying to register the server:

“Error connection to [machine name]”

“A network-related or instance-specific error occurred while estblishing a connection to SQL Server. The server was not found or was not accessible. Verify that the instance name is correct and that SQL Server is configured to allow remote connections. (provider: Named Pipes Provider, error: 40 – Could not open a connection to SQL Server) (Microsoft SQL Server, Error: 1909”

In case you’re wondering why I would want to establish a remote connection to a desktop machine, it’s to run SQL Data Compare against the same project for which work has been alternating between different machines and I want to synchronise the data.

References

First of all, the answers are all out there already, I just had to try a number of them before things began working as expected. Pinal Dave sums up common causes pretty well in this post with lots of illustrated examples of probable causes. Given the Windows 7 / SQL Server combination is probably a pretty common one I thought I’d give a cut down version specific to this configuration here. Both my installations were essentially out of the box configurations (albeit with some components excluded) so my experience should be broadly relevant.

Ensure TCP/IP Connections are enabled

By default, my instance was disallowing TCP/IP connections. Fire up the SQL Server Configuration Manager and make sure the TCP/IP protocol in the SQL Server Network Configuration section is enabled. If it’s not, you’ll need to restart the SQL Service after you turn it on.

One word of caution on this; I set both the SQL Server service and the SQL Server Browser service to require a manual start on the laptop to conserve resources unless absolutely necessary. However if the browser service is not enabled, the TCP/IP setting is lost on reboot. I ended up enabling the browser service (which has a very small resource footprint) but leaving the SQL Server service on manual. Problem solved.

Configure your firewall

This one is probably obvious and disabling the firewall completely (only for testing purposes, of course) was one of the first things I did but it only works if TCP/IP connections are enabled which mine weren’t to begin with. First up, open the “Windows Firewall with Advanced Security” settings (Windows key, “Firewall”):

Click on “Inbound Rules” in the left column then choose “New Rule…” from the top of the right column which will have just appeared.

Choose “Port” as the rule type then continue to the next screen.

By default, SQL Server talks over port 1433 so unless you’ve changed this, enter it into the “Specific local ports” field and continue.

Obviously we want to allow connections over this port so accept the default and proceed to the next screen.

You can choose to lock the rule down to a specific profile but for my purposes I’m happy for it to be accessible across Domain, Private and Public.

Finally, give your rule a name and finish up.

Summary

Pretty easy once you know where to look! Unfortunately for me, once I finally got the machines talking to each other SQL Data Compare told me it couldn’t do a comparison because version 6 doesn’t know what a SQL08 Date type is! Other than that though, it’s been a successful little exercise.

The black art of splitting a Subversion repository

2009-12-12T07:19:00.001+11:00

Here’s the scenario; you have a Subversion repository that has been doing some multitasking. For whatever reason (convenience, laziness, ignorance), the one repository was used to store multiple projects and having now seen the light you want to split it out into separate repositories. This is entirely possible but it takes a bit of work and in many cases, quite a bit of trouble shooting (the kind you won’t normally find in the SVN books). I’ve done this a number of times recently and learnt a lot in the process so I thought I’d capture this info so that hopefully others can avoid some of the pain I’ve been through!

Background

We’ll assume we have a single repository with a folder in the root called “Websites”. This folder then contains multiple projects each in their own folder named after the particular project. For the purposes of this post we’ll assume the path is “ProjectPath”. During the lifecycle of the project it may have been moved around between different folders or even had content from other projects moved into it.

Process

There’s a three step process involved in splitting a repo:

Dump the source repository
Filter the dump to extract the project
Restore the filtered dump to a new repo

In a perfect world we’d simply do all of this in one go but unfortunately it’s a bit more involved than that. It needs to be done in the three steps and it all needs to happen from the command line using svnadmin.

Dumping

Creating a dump of a repo involves svnadmin enumerating through the revisions in the repo and extracting it out into a dump file. An important point to make clear right now; Subversion is very efficient in terms of the repo compression algorithm it uses. A dump file is pure uncompressed plain text (expect of course for any binaries it contains). The bottom line is to ensure you have plenty of free space available on the drive you plan on using for this process.

To do the dump we’re going to use the dump subcommand. The syntax for this looks like the following:

svnadmin dump REPOS_PATH [-r LOWER[:UPPER]] [—incremental]

The lower and upper bands are quite handy if you only want to take a particular range of revisions from the repo. This can make the process quite a bit faster and consume a lot less space so if your project only occupies a small window of time in the repo history then try and use these switches. The particular project I’m going to use has revisions spread out over pretty much the entire revision history so I’m going to leave these out.

Depending on the size of your repo, you could be in for a long wait but at the end of it you’ll have a file called Websites.bak containing your entire repository history.

Filtering the dump

This is the first step where things can start to go wrong. What we need to do is to try and extract just a portion of the dump to create a brand new dump and we can do this by either taking the white list approach and explicitly including paths or the blacklist approach which means excluding paths. Either way, we’re going to use the svndumpfilter command to specify the filter approach, the dump we just created, the dump we want to create from the filter and one more parameter I’ll explain shortly.

When the dump is filtered, there are going to be revisions where “ProjectPath” doesn’t have any changes because it was related to another project in the same repo. The “—drop-empty-revs” switch ensures that if there were no changes to “ProjectPath” in the revision then there will be no records in the dump. It sounds like this will create gaps in the revision history and this is true for the dump file but the gaps will be filled when we load it back in later.

Here’s where is gets tricky; 9 times out of 10 this command will run fine. Where it goes wrong is if your project has moved around a bit. The path named “ProjectPath” might be valid no but what if it was renamed at some time? Or moved to another root path in the repo? Or files from another path were moved into it? Remember the command above only included revisions in the “ProjectPath” folder. Here’s what happens:

Getting the filter right

The skipped revisions are there because we added the drop switch to the command. Further up this screen grab the filter was happily extracting revisions found in the included path until it came across a reference in revision 359 to a path that wasn’t included. This is why we see the “Invalid copy source path” error. What we need to do is add this path so it will be included in the filter.

You may need to go through this process multiple times. I had to do it half a dozen times on a project recently and unless you know ahead of time what all the referenced paths are (unlikely with a large project spread out over a number of years), it’s just simple trial and error to discover them.

Loading the dump

So now we have a dump which has successfully filtered every revision within the “Websites\ProjectPath” folder (and of course, the other paths the project may have occupied before that). The next step is to load this into a brand new repository which in my case I’ve called “ProjectRepo”.

Back to the svnadmin command again but this time with the “load” subcommand. The first thing we’re going to do is to use the –ignore-uuid command. This is important because the unique identifier of the original repo is going to be different to that of the new “ProjectRepo”. Next we’ll just specify the repo we want to load into and the path of the dump file.

What this command is going to do is step through the dump, revision by revision, and restore it into the new repo. You can actually watch this as it goes; open up your favourite repo browsing tool and whilst the load is running you’ll see the project being gradually reconstructed in chronological order with sequential revision numbers (no empty revisions). Until you hit another error…

Fixing the dump

This is the second point where things can start to go awry. Most times the load will be a smooth process but as with the missing paths error above, a repo that has moved around a lot may well end up with errors.

Let’s look a bit closer at this error: “File not found: transaction ‘0-1’, path ‘Websites/ProjectPath’. To understand what’s going on here we need to get our hands dirty and start looking inside the dump file.

Because the dump file is uncompressed and plain text (except for any binary content, of course), it’s easy to open it up in a text editor (I usually use Notepad++). When you do you’ll notice the file is broken up into a series of commits, each with it’s own revision number (line 15 in the image on the right), commit message (line 22), author (line 26) and then a series of paths and actions.

The problem lies in the very first revision. What’s happening is that when line 33 is being executed, the path “Websites” does not already exist and this is why we’re getting the “File not found error” above. This transaction worked just fine in the original repository because the path had already been created in a previous revision but because the filter ran at a level lower than this the transaction which created the folder wasn’t included in the dump.

The fix is easy; we just need to rewrite history! This is actually a good thing because it will allow us to get the repo structure right. Ideally, the work should be in a folder called “trunk” at the root of the repository. What we’re going to do is change every occurrence of “Websites/ProjectPath” to just “trunk” with a quick find and replace. This way when the revisions are imported to the repo it will appear as if the structure was correct from day 1.

Rinse, lather, repeat

The missing paths problem is one which may occur multiple times in the repo. Unfortunately you don’t know how often or where until you run through the load process again and it fails. Each time you need to go through and fix the dump file as per the process again until everything runs without error.

Summary

This all ended up being a lot more difficult than I originally expected, largely because of the unexpected errors which continued to crop up, the volume of trouble shooting required and the long durations to dump, filter and load a very large repository. Lesson of the day; think very carefully about your repository structure early on because it can be a serious headache to reorder later on.

I’ve been undefined!

2009-12-08T21:38:00.001+11:00

Just in case anyone has passed by in recent days and noticed the Twitter panel on my blog displaying tweets that are, well, inconsistent with my normal tweets, fear not! It seems the geniuses behind the Twitter webpart for the Blogger engine have made a small mistake in the JavaScript variable department.

In short, it appears an undefined variable is somehow inserted into a JSON request URL to Twitter and unsurprisingly, the Tweets of an undefined user are being requested. Surprisingly enough though, someone has actually decided that “undefined” is, for some reason, a sensible enough Twitter username to actually use on what appears to be a fairly active basis.

So here we are; undefined (AKA “Adam” based on his website) has unintentionally hijacked my blog’s Twitter feed! From his recent tweets it seems pretty obvious Adam has received quite a bit of attention from people confused by his sudden appearance on their blog. After all, the Blogger gadget in question is the one Google recommends for redisplaying Twitter feeds so I imagine there are a large number of people, inevitably often not technical, who are wondering what the hell is going on. It’s also clear based on Adam’s tweets he’s having a lot of trouble getting through to Google.

Here’s how to fix it

Easy; get rid of the Blogger gadget. This has happened before and with the issue surfacing again I’m not confident it won’t happen again. So let’s keep it simple and ditch the gadget. Fortunately the folks at Twitter have provided a very nifty mechanism for adding a Profile Widget to your website which is configurable via a browser based UI. Go through the steps, preview the widget then there’s an even an easy option to drop it directly into a Blogger based web log. Problem solved!

For the record

Just in case someone from Google does finally decide to get proactive and fix the problem, here’s the Fiddler trace of what’s going on and where it all goes wrong. I haven’t dug any deeper into the source of the problem but a 404 on the .js in request 13 followed by an undefined variable seems very coincidental:

Building the ultimate virtual office

2009-12-06T19:01:00.001+11:00

I’m fortunate enough to work in a role which allows me to spend of lot of time talking to people in all sorts of places around the world on a daily basis. As much as I enjoy this varied communication, it plays absolute havoc with your working hours. It’s not so bad speaking to folks in the Americas (I’m a morning person!) and that works well for them but throw in a country somewhere between Sydney and New York and you always end up with a pretty inconvenient time for at least one or two parties.

All this has led to me spending a lot more time taking calls from home in part because of the odd hours but also because it allows me to work in an environment without the usual disruptions you get in an open plan office. On top of the usual work calls and out of hours business, I’ve found myself spending a lot more time researching and playing with emerging technologies lately, all of which has me spending more time in this environment. Given this, I decided to set out and build what for me is the ultimate virtual office. A lot of time and research went into getting everything setup just right so I thought I’d share the results in the hope that other people might find it a useful reference.

Environment

The first thing is I’m fortunate to live in a house that has a lot of space. I’ve got an attic area which is not part of a thoroughfare to other parts of the house and once the door is shut, is isolated from anything else going on downstairs. Having lots of space is not essential to a virtual office but it sure does give you more options when you come to designing one.

Research

Firstly, there are a heap of opinions out there on what works well and not surprisingly, not everyone agrees with each other! Having said that, I did start to see a couple of trends which have ultimately formed the cornerstone of my own virtual office. Some of the resources I found really interesting included a Shed Turned Home Office, Scott Hanselman’s Home Office and this pretty extreme example. See any similarities with these? Two things; lots of screen real estate and a very cool chair. In fact exactly the same chair in each example.

Chair

This might not seem like a logical place to start, but the chair is quite possibly the most important item to go into any office, virtual or otherwise. Beyond just the fact that you spend a considerable amount of time sitting on it, the chair goes a long way to determining your health (particularly when it comes to RSI), your comfort and consequently your productivity. The other thing is the chair has possibly the longest lifespan of just about anything you put in your office. Having said that, it’s a question of quality. A cheap item from Ikea is not only going to expire pretty quickly (my previous one got a few years before it started falling about) but it’s also going to be pretty uncomfortable in the process.

This is where the Herman Miller Aeron comes in. This is as much a piece of art as it as a functional piece of furniture (Wikipedia actually says there’s one in the New York Museum of Modern Art) and it was a symbol of exclusivity during the dot com boom. Interestingly enough, I’m actually seeing a lot of them now on TV shows, being used by news readers and even by the receptionists in the office (not sure why we ended up with simple plastic ones…).It’s an extremely solid piece of furniture with many components such as the arms on the base of the chair being solid steel and not the usual plastic. Even the arms on the arm rests are solid and topped off by nicely padded, adjustable (everything's adjustable!) rests which are very comfy.

The downside with these is that they’re not cheap but it really is a matter of getting what you pay for. Given the amount of time I spend in it and the lifespan I expect it to achieve I decided to take the plunge and dive in. You can get them in a few different sizes to suit, shall we say, different “girths” and although the medium was a snugger fit I ended up going with the larger one which allows me to really sink back into it and even comfortably cross my legs on the chair. The beautiful thing about this chair is that no matter which way I lean or how I position my body it’s always comfortable. I can happily sit in this all day long without getting sore or cramped which is not something I can say for any other chair I’ve used before.

Couch

Ok, this is by no means mandatory office furniture but it has become a lot handier than I expected. It gives me the opportunity to change my working style without actually leaving all the office facilities or becoming distracted by external factors. I love being able to take the laptop over with me and work through a backlog of emails or get engrossed in typing a document whilst I’ve got my feet up. It’s just perfect for when you need a change of environment without breaking from the focus an office environment gives you.

Monitors

There’s an easy rule of thumb here; the more and the bigger the better! Widescreen LCD monitors have gotten very cheap recently and the sweet spot of value versus features is going up pretty quickly. I found the 24” models are right on the money at the moment, so I got two. Other than the size and the price, the main thing that was important to me was height adjustability. I didn’t want to end up needing to jack up the height with monitor stands or arms; it just needed to work out of the box.

The other really critical thing is that it needed to support 1920x1200 resolution. The resolution is really important because at 24” you don’t want to be looking at massive pixels which is where you’d end up at traditional resolutions. Just to put that resolution in perspective, we’re talking about 2.3 million pixels multiplied by two monitors which is the equivalent of having six traditional 1024x768 monitors.

Samsung have got a pretty good range of monitors which are not only height adjustable but also have very good ratings for response times, brightness and contrast so I ended up getting two 2443BW models. The monitor also includes a built in USB hub which I don’t personally use but it’s always handy to have. The whole screen can also rotate 90 degrees which I’m sure could be very useful in some contexts but I don’t expect I’ll have a need to use it in the mode. Like most LCDs these days there are inputs for both VGA and the digital HDMI format and the box also included a VGA to HDMI converter so plenty of connection options are available.

The next thing was making sure both monitors were usable simultaneously. There are heaps of studies and articles out there about multiple monitor productivity and it’s becoming unheard of for software professionals to just scrape by with a single screen. The trick is making sure the PC hardware can make the most it. My home PC was straight forward as I have a video card with two HDMI out connections so it was just a matter of plugging them in and letting Windows work it all out. The laptop was a little trickier though. In the office I use a docking station which has both a VGA and an HDMI out so I simply plug both those in to the two monitors at work but I didn’t want another docking station to be “docked” at home.

Fortunately Matrox has a really neat solution called DualHead2Go which is a little hardware device with a single VGA and a USB input and two HDMI outputs. This device can output two video streams of up to 1920x1200 each by tricking the host machine into outputting one massive video image (dependant on graphics card and driver version) then splitting it in two before sending the signals to the monitors. The USB connection then uses bundled software to give the machine a sense of what size each window is so you can do things like maximise your windows just to a single screen. Finally, the device can actually output the two 1920x1200 streams plus still use the built-in laptop screen of the same res so you can effectively have three screens totalling 5760x1200.

Desk

There’s really not a lot of technology that goes into a desk. The success Jeff Bezos has had using a door as a desk is proof you can reach great heights without being high tech! For me, the primary criteria was to have enough space without feeling cluttered. I needed to fit a couple of sizeable monitors, my laptop, phone, multifunction printer and have usable space for keyboard and mouse and still feel like I had plenty of free space. I was also aiming for a very light and airy feel so in the end I went for a white enamel item which included a primary desk, a return and some drawers. The return has actually worked out very well as it gives me a place to sit the laptop and use it simultaneously with the desktop.

Silent PC

I didn’t realise how much of an issue fan noise was until I began rearranging the office and moved my home PC into a location that that directly above our bedroom. The low frequency vibrations that fans put out tend to resonate through floors and become quite audible while lying in bed so something had to be done about them. In a quest to remove and replace the noisy culprits I ended up replacing the two 80mm fans on the front of the case, the 80mm one on the back and the entire power supply. After all that, I accidentally snapped a pin on the CPU when trying to replace the heatsink fan and to cut a long story short, I ended up in a vicious cycle of replacing interdependent components (or others that were just too old) until I had an entirely new motherboard (with no fan this time), CPU (with a super quite, gigantic heatsink fan), graphics card (with no fan) and RAM. Basically everything but the case and the drives!

What I learnt about making a PC silent was quite interesting. There’s an entire site called Silent PC Review dedicated to the art of eliminating noise. The crux of making a PC silent is to use larger, lower revolution fans with high quality bearings and to use dampers between any moving devices and the mounts to the case. To that effect, when I ordered the fans I also order siliceous sheets which are essentially moulded pieces of silicone which sit between the fan and the case and act as little dampers. The Noctua fans I bought also came with rubber mounting pins instead of the usual plastic ones so noise or vibration transfer through to the case is absolutely minimised.

Speakerphone

One thing I do a lot of is talking on the phone. Unfortunately not so much in the social sense but very frequently on teleconferences and often for an hour or more. Handheld phones are simply too uncomfortable for this style of working, especially when meeting participation can often be a passive task and you want your hands free to do other things. The other thing is you really need a mute button out of courtesy to other meeting participants so that when you’re not talking you can type, answer your mobile, have noises going off in the background or whatever else without being an inconvenience to everyone else.

I looked at a lot of different speakerphones but kept coming back to the Polycom Soundstation. You’ve probably used these in meeting rooms before and they’re almost the de facto standard for voice conferencing. Unfortunately they’re not cheap when they’re new but there are plenty of them on eBay and I managed to pick one up for not much more than a tradition style phone with speaker capabilities.

VOIP communication

If you’re not using VOIP in some fashion then the chances are you’re paying too much for phone calls. As much as I like the idea of VOIP though I don’t like the idea of being tied to my computer nor do i like the idea of having one phone for landline (I still want to at least receive phone calls over PSTN) and another for IP based discussions. Fortunately there are a few “dual phones” out there where the base connects to your analogue phone line and either to your PC (which then runs a client) or directly to your wireless router. I ended up getting some cordless Dual Phones which consist of a base then several satellite phones with charging bases you can distribute around the house. As it turns out, they’ve all broken (yep, all three of them) but the concept is very good and I’m sure I’ll find another hardware manufacturer that does a decent job quality wise. Great concept, just poor execution by this particular manufacturer.

Ergonomics

Most people (myself included) only begin to think about ergonomics after they begin suffering from working in an ergonomically unsuitable environment. I began getting tingling in my fingers which would then go all the way up my arm and become quite painful. I ended up getting some physiotherapy which included giving me exercises and stretches which helped but the biggest change I made to my working style was to get a decent keyboard and mouse both in the office and at home. These are such simple, comparatively cheap devices and they can make a huge difference to the way you work. If you’re like me and regularly spend the whole day (and often night!) in front of the computer then take a look at the Microsoft Ergonomic Desktop 7000. This is widely considered the equipment of choice by those in programming circles and has almost completely reversed my RSI symptoms.

Audio

More than just making beeps and alerts, I use my audio setup pretty extensively for watching video content and very regularly for listening to music. This isn’t everyone’s cup of tea in an office environment but I like having the option to pump out something that suits my mood. I already had a Logitech Z-5500 which is a pretty typical high-end PC setup. One thing I really like about it is that it has an optical in which I can drive from the PC plus a standard analogue audio jack I can plug into the laptop or iPhone as required without disrupting the PC setup. You can then select the input source as well as control the volume from a desk mounted control unit which is really neat.

Multi-function printer

These are literally a dime a dozen now and are very handy little devices. The Canon PIXMA MX700 I got was only a couple of hundred dollars and is a colour scanner and copier (both with automatic feeder so you do stuff in bulk) as well as a fax and a colour printer. The other really neat thing about it is that it’s network attached so there’s no dependency on sharing it through a PC. It’s just always there and always on independently of any other devices.

The only complaint I have of it is that for some reason I can’t set the fax to allow more than five rings before answering which is leading to a lot of running around the house when the phone rings (shared phone / fax line). I think I’ll just bite the bullet and call tech support or even just unplug the phone line; who sends faxes these days anyway?!

Cable management

This is going to sound like I’m a bit fussy but I hate it when you end up with cables all over the place. At last count I have 18 separate devices all plugged into power and easily double that many other cables so there is a lot going on under the desk. The easiest way I found to get everything properly organised was to buy a few cable trunking devices from Ikea which you can then mount up under the desk and run the cables through. It’s a neat design with lots of little wire “fingers” sticking out from the edges so you can drop cables off or join them back to the trunk at any point. Once it’s all set up it means you can pretty much secrete everything up under the desk and not need to look at it on a daily basis. Of course if you get under the desk and see what’s going on it’ a bit of a nightmare but I tend to avoid doing that!

The end result

In the end the whole process took longer and cost more than I expected but I’m very happy with the result. I’ve now got an environment I can comfortably sit in for hours on end in peace and quite with everything I need to be super effective right at my fingertips. My biggest problem now is making sure I make a conscious effort to get out of the office and enjoy the beautiful Sydney weather!

Here’s how it all turned out:

PDC09 Roundup

2009-11-22T10:48:00.002+11:00

Unfortunately the tyranny of distance has kept me well and truly away from Microsoft PDC09 but it hasn’t meant I’ve had to miss out completely. Seeing so much exciting information released about new technology (which happens to be my favourite kind!) made me want to get what excited me down online.

Silverlight 4 Beta

In case you’re feeling like it was only yesterday that Silverlight 3 was released, you’re almost right; it was back in July. The velocity of this product is like nothing I’ve seen from Microsoft before.

In terms of Silverlight 4, PDC marked the launch of the first beta. There’s a full rundown on the silverlight.net site but here’s what jumps out at me in terms of things I’d find useful:

Comprehensive printing support (about time!)
Lots of new controls including a rich text box (wonder what the markup language is?)
WYSIWIG support in VS2010 (ok, more a VS feature but will save some Blend / VS switching pain)
Elevated privileges for out of browser (big stuff in terms of building apps which need more machine interaction)
Access to devices such as webcam and mike (part of the goodness elevated privileges bring, see a demo and code sample here)
Implicit styles to theme the UI (think CSS and defining styles by tag name rather than by class attribute on the control)
Cross domain access without security policy files (will make getting access to external services much easier)

Apparently Silverlight is also now on something like 45% of machines (I know, a tweet is not a very authoritative source but it’s Nikhil and it sounds feasible). Tim Heuer has an excellent rundown including demos and pictures in his post Silverlight 4 Beta – A guide to the new features.

IE9

There’s An Early Look at IE for Developers on the IE Blog but in short, performance enhancements and greater capability are the key themes:

Much better alignment to Acid test 3
More implementation of CSS3 (including features like rounded corners)
Performance enhancements to the rendering engine and use of DirectX

Reading through the comments on the blog, the recurring theme is “Great, yet another browser to make my sites compatible with” (just the natural fallout of browser evolution I’m afraid). Webmonkey is also reporting Microsoft Still Chasing the Competition With IE9 which is reflective of the general sentiment I’m seeing – too little too late.

iPhone

No, it’s not a typo, Microsoft actually used an iPhone in a demo without the intention being to denigrate it! They used it to highlight IIS Media Services by embedding the video tag from HTML5 and pulling the data off IIS. Take a look at it on an iPhone here. I had a go on mine and the quality was sensational.

If you’re not already familiar with the proposed HTML5 spec, Wikipedia has a brief spec overview plus a detailed overview of the support by layout engine which shows wildly varying compatibility levels (remember though, HTML5 is still just a spec). Early days but at least gives us some more options when targeting iPhone.

Windows Azure

Get used to reading about Azure because you’re going to be seeing a lot of it in the coming months. Its been in CTP for the last year and is Microsoft’s interpretation of what cloud computing looks like. The entire cloud space is heating up pretty quickly with Microsoft competing against Amazon EC2 and Google App Engine, each with their own unique implementations.

PDC saw Microsoft finally reveal the pricing model which starts at US$0.12 per service hour. There was a bit negative feedback about the cost on the Twitter-sphere but when you consider you’re talking about less than $3 a day (c’mon, I spend more than that on cappuccinos!), it doesn’t seem like very much at all. The inevitable comparisons to how this stacks up against existing hosting providers will happen but Azure and cloud computing in general is about a lot more than just hosting a website.

The other Azure news we saw out of PDC09 was “Dallas”, a service enabling developers to discover and subscribe to data feeds. Here’s how it’s explained on the ars technica blog post:

The technology was showcased by PDC by Federal CIO Vivek Kundra. Kundra demonstrated a career-finding application based on Department of Labor teaching data stored and catalogued by Dallas that allowed, for example, teachers to find which areas of the country needed more teachers. The application was able to drill down within the dataset, for example, to find out exactly what kind of special education teachers were required in a particular area.

Obviously the dependency here is in providers making their data available so hopefully Microsoft can make it a compelling enough proposition for the service to be attractive to data consumers. Probably a “chicken and egg” scenario here so I do wonder how quickly providers will come on board.

I’m genuinely excited about Azure; if Microsoft can make the offering attractive enough to corporates by fulfilling on the promises of security, performance and utility based pricing and deliver to developers by streamlining infrastructure provisioning and app deployment within a familiar coding environment, they might just be on a winner.

Seesmic

Ok, not a Microsoft product but Seesmic got enough Twitter coverage to justify a look. In theory they make a Twitter client. I say in theory because all I got when i tried to use it was the little guy in the image below. Full marks for an amusing error message, no idea what the tool is actually like though!

Free Laptops!

It’s of zero direct benefit to those of us who weren’t there but those who made themselves available were rewarded with a free laptop. Reading the info on the Channel 9 website, it seems the not-so-hidden agenda is to get developers using Windows 7 and multitouch which without dedicated hardware would be extremely difficult, at least from the perspective of experiencing your product firsthand in the intended environment. Given this significant show of multitouch evangelism from Microsoft it will be interesting to see where they’re going with this product.

Next steps…

Personally, it’s Silverlight 4 and Azure which have me really excited. Silverlight because of the potential to break free of the shackles of HTML and change the way we build applications whilst overcoming some of the SL3 and earlier barriers, Azure because of the radical way in which it could address so many of the headaches developers face in terms of provisioning, hosting and supporting applications. It looks like there’s going to be plenty of reading and experimenting to be done over the Christmas break!

Is software quality really important? Ask Virgin Blue…

2009-11-14T11:32:00.001+11:00

Often times in the software industry you’ll hear statements suggesting that the quality of the underlying software code is really not that important and what matters most is immediate term customer satisfaction, primarily achieved by delivering what they asked for within time and budget. This is usually a position put forward by the customer facing folks within the software industry who, in their defence, probably don’t have that strong a grasp on why code quality is important simply because they’re not living and breathing it. So let me give you an example.

Gold membership!

Yesterday I was both surprised and delighted to receive the following email from Virgin Blue, our local Australian arm of the Virgin airline empire:

As excited as I was about the prospect of receiving not one, but two personalised luggage tags, I was a little surprised that my meagre two short haul, domestic trips would qualify me to jump a couple of status levels in one go. But hey, who am I to complain?!

If something sounds too good to be true…

A mere two hours later the truth was revealed. I didn’t qualify for an upgrade and I would not be receiving my personalised luggage tags. Not even one of them. As it turns out, the root cause of the problem was due to the day of the week being a Friday and the day of the month being the 13th. Right, nice try.

Digging a little deeper

It turns out someone screwed up big time judging by the breadth of the problem as reported across the Twitersphere. The Australian IT News website reports Virgin Blue "error" upgrades passengers to gold status and explains the issue as follows:

The Velocity website appeared to crash at 6:40PM AEDST Friday, hinting that a large number of customers had visited the site to find out if the upgrade was true.

and then

A call centre representative from Virgin Blue said that the email was “a system error”.

“IT have advised that we do have a system error,” the representative said.

“Right now we do have a lot of phone calls because of this error."

And this brings me right back around to the topic at hand; software quality really is important. The thing about software quality is that often times it only becomes a priority once it’s too late and the pain of financial loss and public humiliation is felt.

Counting the cost

Let me take a stab at how this has hit Virgin Blue in the hip pocket:

Their website went down so they lost bookings
Their call centre was tied up dealing with unhappy customers so they lost even more bookings
IT staff and management were kept preoccupied in “damage control” rather than doing their normal job
A carefully managed corporate image (and the Virgin brand is very carefully managed), has been tarnished

It’s the last point that really hurts. Virgin has constructed such a carefully honed brand image as an organisation that really relates to people, dangling a carrot in front of their customers then taking it away in the blink of an eye is very, very bad for business.

What went wrong

I can only speculate, I honestly have no inside knowledge of what happened within Virgin’s software yesterday. What I can do is take a pretty educated guess at the conditions which lead to this after having spent a lot of time seeing the inner workings of software products.

Assuming the content of the email was correct in that some customers who would not normally be eligible for Gold but were “close enough”, there is inevitably some conditional logic along the lines of “if customer points balance is > 90% of points required for gold then upgrade”. There may even be a couple of conditions such as “and > 6 flights taken this year” or “and has been a member for > 2 years”. Now imagine a “>” was accidentally replaced with a “<”, or the “and” condition accidentally became an “or” condition. Easy to do and every software developer has done it before (or they lie and say they haven’t). The difference is that this went through to production unchecked and caused Virgin a big headache.

The point is, the conditions required to legitimately qualify for this offer were probably pretty simple and were equally simple to get wrong. The issue also occurred just before 5pm on a Friday night so picture this situation; its been a long, hot week, everyone’s about to head out for beers, the office is emptying and there’s just this one little task that needs to go out the door beforehand…

Of course I’m speculating but without any information from Virgin to the contrary, I think they’re pretty fair assumptions. Regardless, it’s safe to assume we’re talking about a software quality issue here.

Because quality does matters

There are probably a number of different ways this could have been avoided by looking at software from a quality standpoint. Perhaps there were no automated tests for this particular condition. Perhaps an activity with such broad ranging ramifications didn’t alert the operator to the scope of the audience. Perhaps the logic was embedded in a 500 line file and indented through half a dozen “if” statements hence obfuscating its true intent.

Whatever it was, someone skimped on quality and now the true ramifications of this have come to bear. So the next time someone attempts to convince you software quality doesn’t matter, ask them if they ever received their personalised luggage tags. Hopefully the raw emotion and bitterness of their loss will help them see the error of their ways!

Want to be a better programmer? Have a baby!

2009-11-06T08:09:00.001+11:00

Recently I’ve gone through a particularly intensive training program. This program has required a significant contribution of money, time and energy and although it has already been running for four weeks, has only just begun. The thing is though, its already made me a better programmer and the course is pretty easily accessible, in fact some people even stumble across it accidentally; get yourself a baby!

What the?!

Ok, I know it’s a nonconventional approach but allow me to elaborate. The journey into parenthood forces you to push all sorts of boundaries you wouldn’t normally consider. It requires degrees of both creativity and patience you’ve probably never experienced before. It also introduces you to the concept of Baby Brain (I assure you, women do not have a monopoly on this phenomenon!) but for the most part, it’s an eye opening experience.

Problem solving

Programmers tend to be very adept at solving problems. In fact, you could say our entire existence is dependent on being able to identify, analyse, design and build solutions for problems. Generally we proceed in a very analytical fashion, carefully analysing the problem by identifying the current state, looking for patterns then constructing precise solutions.

One thing that pretty quickly becomes clear about babies is that the usual rational problem solving techniques go right out the window. Think you can implement the same solution twice and get the same result? Think again. Think you can identify the root cause of just about any problem? Not likely. Try brainstorming with a baby – it just doesn’t work.

When you start troubleshooting a baby you enter a whole new world of problem solving pain. You develop skills which take you into totally unchartered mental territory and force you to think not only outside the square but outside any semblance of a normal, logical human behaviour. I’ve actually had customers like this in the past and now I know I’ll be far better equipped to deal with them next time.

Becoming a better multi-tasker

One thing you learn very quickly is that life becomes more multi-threaded than ever. As well as all the usual information you absorb and process on any given day you suddenly need to spawn a dozen new threads to deal with such important equations as “when did he last eat?” or “what exactly was in that last nappy?”. Life becomes parallelised to a whole new level.

For a programmer, this a very good thing. Developing the ability to simultaneously juggle a broad spectrum of tasks of extremely diverse nature is a desirable trait. In fact people pay very good money to learn how to do this effectively (quick tip; save your cash and read Eat That Frog instead), when all you really need to do is conjure up a baby and “hey presto” instant multi-tasking ability to rival the best of them!

Discovering new things about technology

The whole baby managing process ultimately means familiarising yourself with a range of foreign objects and practices. Fortunately, the connected age means you can often do this with technology and introduce a bit of baby geek factor into the experience and discover a few new things along the way.

Baby monitors; they seem pretty simple in concept and for the most part they are unless you get one of those new fandangled ones that includes things like a breathing or movement monitor (which we didn’t). What you don’t learn until a little bit later is that they also tend to mess with your wifi. You learn this after many router resets, channel changes and stomping around the house moving potentially interfering objects.

Here’s another interesting fact I picked up along the way; your iPhone can serve content via HTTP to other devices. This is probably old news for many people, but if it wasn’t for us using Total Baby and wanting to backup the data I never would have discovered this. The app provides a basic HTML interface served up over IP (which is kindly provided by the app) so can just download the data and store it elsewhere. Neat.

Seeking out sane, adult conversation

Maybe it’s just my short attention span, but day in day out discussion about nothing other than bowel movements, breast feeding and sleep patterns leaves you in dire need of “normal” conversation. In my case, this was a perfect opportunity to soak up as much techie related information as possible which meant spending a lot of time reading blogs and Twitter.

The irony is not lost on me; saying I was seeking “sane, adult conversation” then in the same breath suggesting Twitter is a viable source for this material is just plain ridiculous. Desperate times, desperate measures! The thing is though, I’ve absorbed an abnormally large amount of information during this time. I’ve followed people I wouldn’t normally have thought of, I’ve read articles I would have previously discarded and as a consequence, I’ve gained a whole bunch of actually quite useful knowledge. All this stemming from a vacuum of sane, adult conversation.

Loving your day job

I can’t recall the last time I relished going back into the office as much as I did a couple of weeks back when returning from leave. Don’t get me wrong, it’s not that I enjoy being away from my family, that’s not it at all. The cause of my excitement was being able to sit down at my desk, in peace and quite, for as long as I wanted without having to clean something, attend to something or burp something. I’ve found a totally new love for the workplace!

Embarking on this course of self improvement

Embarking on this particular course is quite straight forward to begin with (Google it if you’re not sure!). However, it’s not for the fait hearted and there are no refunds if you decide to drop out halfway through. In fact, I understand there are some pretty severe penalties so you want to be committed upfront. However, I do love the non-conventional and as far as alternative study goes, this has got to be right up there with the best of them. I may consider taking a second course in the years to come but for now, I’m dealing with about all the self improvement I can handle!

25 illustrated examples of Visual Studio 2010 and .NET 4

2009-10-27T19:25:00.001+11:00

In my quest for both some engaging blog material and a desire to get up to speed with everything new in the latest generation VS and .NET fast, following is an overview of 25 illustrated examples of Visual Studio 2010 and .NET 4. It’s not seriously in depth, just enough to understand what’s new and where you can go to get the serious details.

Background

Rather than churning out another ubiquitous “Hello World” app, I’ve used an existing ASP.NET 3.5 project which has been my test bed for pretty much every version of .NET released so far. Hopefully this will give a bit more practical context than something hypothetical although it does mean a webform bias to the post. The features below are in no particular order, they just appear as I’ve discovered or read about them.

1. Starting up

The first obvious difference with VS2010; no more rectangular start up screen! Ok, it’s of zero functional value but it does give the product a pretty swish new look.

Actually, the first big shock was no more Mr Tiny Face during the install. Maybe it was the public ridicule, maybe it was just inconsistent with the new branding but either way, I’m going to miss the little guy!

2. Windows 7 taskbar access to recent solutions

This neat Windows 7 feature is now surfacing itself through a number of applications and whilst it’s a bit trivial for programs opening small, fast files (such as Notepad), it’s really handy for Visual Studio. I’ve tended to always have it auto load the last solution on open then choose a recent solution after that so you end up loading multiple projects just to get to where you want to be. Pin it to the Windows 7 taskbar, right click and it’s always a one step process.

3. Conversion process

Pretty much the same deal as in VS08 here with loading an existing solution from an older version prompting a conversion. Any web projects allow you to automatically upgrade them to .NET 4 whilst other projects, such as class libraries, need to be done manually after the solution conversion.

In terms of actual changes to the solution performed by the conversion process, the .suo gets the VS version incremented and the .csproj files update the .NET version and include a reference to the version they were upgraded from. There are a few other minor framework related changes but the biggest move is in the Web.Config which shrunk by more than half after the conversion.

4. A cleaner Web.Config

Scott Guthrie has published some good info about Clean Web.Config Files so I’ll just give you the brief version. In short, a lot of the bulk is moved out to the machine.config and what you’re left with is just enough project specific info to make things run. You also get a reference to control rendering compatibility being set to the previous version of the framework so you don’t get breaking CSS and HTML changes (more on that later).

5. Better multi monitor support

I’ve said “better” support because there was multi monitor capability in VS08, it’s just that you were limited to basic operations such as docking some of the windows (properties, solution explorer, etc) in a second monitor. In VS2010, you can take open file windows and move them to another screen or rearrange them in the same window which is really handy if you wan to see both ASPX and codebehind at the same time. It’s a still a little clunky though; windows don’t open where you left them and it’s easy for newly opened files to appear behind undocked windows. There’s also no ability to automatically arrange windows and they won’t snap to each other or to screen edges so there’s a bit of fiddling required to get an optimal setup.

6. Code snippets

This is a small one but it will save a lot of keystrokes over time. There are now a number of built in code snippets for ASP.NET, HTML and MVC. What it means is that intellisense will pick up you’re possibly looking for a RadioButtonList if you begin typing “<rad” (yes, without the “asp:” prefix) and it only takes the tab key to then automatically insert the entire tag including sample children. You can then tab between the highlighted values for quick editing.

7. Optional parameters, default values and named parameters

Rather than creating a series of method overloads to cater for different scenarios requiring different parameters, they can now be specified as optional by designating a default value (so long as they appear at the end of the method signature). To avoid ambiguity between multiple optional parameters, they can be referenced by name within the method call. To top it off, VS2010 surfaces optional parameter names through intellisense which makes discovery dead easy. David Hayden has a more comprehensive overview here.

8. Dynamic language support

The concept of a dynamic language, one that executes at runtime without knowing how it will behave during design or compile time, is not new to programming (Ruby and Python are just a couple of existing dynamic languages), but it is new to .NET in version 4. One advantage this gives us is the ability to use the DLR (Dynamic Language Runtime) to facilitate communication with systems where strongly typed language features are not available without the bloat of using reflection. This is a pretty simplistic (and possibly slightly inaccurate!) overview though so take a look at Scott Hanselman’s excellent explanation.

9. URL routing in ASP.NET

This was introduced back in ASP.NET 3.5 SP1 and is used extensively in ASP.NET MVC but is still worth a mention here as it becomes a native part of the framework. Routing provides a means of making a request to a URL which doesn’t necessarily map directly to a physical file. Instead, it provides us with querystring like features but without the explicit name / value pair formatting. What this leaves us with is the ability to create very clean, human readable, search engine optimised URLs.

Process wise, it’s just a matter of registering the route in the Application_Start event in the Global.aspx.cs. My aim in the following example was to take the path normally expressed as /Leases.aspx?InvestmentPropertyID={InvestmentPropertyID} and instead allow /Leases/{InvestmentPropertyID}. Doing this with GUIDs is not great legibility wise but the concept translates just as well to strings which would result in nicely legible URLs.

The above code simply gives the route a friendly name, specifies the URL pattern then states the page we want to send the request to. All we need to do then is to read the parameter from the target ASPX page (note: I’ve used a nullable GUID due to the nature of the app not always having the ID available).

Quick note; if you’re trying to make this work with IIS6 or IIS7 running in classic mode, have a read of Using ASP.NET MVC with Different Versions of IIS. If you don’t follow this guidance you’ll end up with a lot of 404s.

10. Intellisense gets more intelligent

This one has been around in ReSharper for ages, but intellisense now filters the available options as you type rather than just displaying all possible matches and defaults to the best match to the characters typed. Where it goes a little further than ReSharper is by not just listing matches that start with the characters but matches which contain them. And if you’re not real keen on a filtered list, CTRL-Space will give you back the full version.

Another really neat feature is matching by acronym where a type or member adheres to Pascal case (also currently supported by ReSharper). If you enter a string in uppercase you’ll get a filtered list of all possible matches (i.e. Grid.DS will give you the option of Grid.DataSource). It will be interesting to see how ReSharper 5 responds to these changes given previous versions has replaced the native VS intellisense altogether.

11. Geo-location awareness

There seems to be very little available information on this so far other than this extract from the BCL Team blog:

System.Device.Location (which can be found in System.Device.dll) enables .NET applications running on Windows 7 to determine the current geo-location (e.g. latitude/longitude) of the device. Windows 7 supports a number of different location sensors, including GPS devices and WWAN radios, and automatically handles transitioning between different sensors — if multiple sensors are available — to provide the most accurate data for the current situation. .NET applications will be able to use this new API to access the following data (if available): latitude, longitude, altitude, horizontal and vertical accuracy, course, speed, and civic address (i.e. country/region, state/province, city, postal code, street, building, floor level).

That’s enough information to get me excited! Coupled with online mapping services, address resolution and SQL08’s spatial data capabilities, we’re getting some pretty cool geo-specific capabilities these days. Here’s a quick glimpse of the members:

12. Code navigation

Scott Guthrie has a comprehensive blog post on the new navigation features and he starts off with an interesting statement:

In usability studies we’ve done, we typically find that developers spend more time reading, reviewing and searching existing code than actually writing new code.

It would be hard to argue with this and Microsoft have obviously taken it pretty seriously. A quick CTRL+Comma now gives you a “Navigate to” window which filters by keyword and includes the type of match (event, class, member, etc) and the file and line in which it appears. As with the intellisense improvements mentioned above, you can also enter acronyms which match Pascal case names within your solution. This feature will come to the forefront in many cases which would have otherwise required a normal “Find” and stepping manually though results.

13. Meta tag properties

A small but worthy addition; meta tag properties are now part of the Page class and can be assigned to directly by reference. Should make search engine optimisation that little bit easier, particularly for data driven pages.

14. Guid.TryParse

To get an idea of how handy this will be, take a look at the number of Google results for “Guid.TryParse” that don’t refer to the .NET 4 implementation and instead refer to either a desire to add this to the framework or take you to people’s own bespoke implementations.

15. Call hierarchy

There’s more than just a passing similarity to ReSharper’s “Find usages” in this feature but it extends the concept into some new territory. The call hierarchy feature not only displays usages (or “Calls To” as it’s phrased"), but also calls it makes (“Calls From”). There’s also a drill down feature so you can navigate through the tree of nested calls which is a pretty neat way of quickly understanding precisely how the code is being used.

16. Auto-start for ASP.NET applications

How many times is someone left wondering why an an ASP.NET app is running so slowly only to realise it has been rebuilt or the application restarted and there is yet to be a client hit to the app? Certainly happens to me a lot so I’m pretty keen on the new auto-start feature.

The only downside to the feature is there is an IIS 7.5 dependency so unless your web server is pretty current, you’re not going to get any value from it. If you are current, it’s just a simple change to the applicationHost.config file to set the startMode attribute to “AlwaysRunning”. More info, once again, from Scott Guthrie here.

17. Usage highlighting

This is another feature previously surfaced through ReSharper using Shift+Alt-F11 (are you sensing a theme here?!) but happens automatically in VS2010. Select a name anywhere within code and all usages are automatically highlighted. Just a little feature but yet another small productivity boost.

18. Managing the client side ID of controls

Anyone who has ever written client side code to reference server side generated controls will appreciate how much easier this can make life. Rather than the auto generated IDs like ct100_BodyPlaceHolder_TenantID we can now utilise the ClientIDMode to achieve greater naming control. This can be configured at the page level, in the config file of either the application or the machine or at the control level. Take a look at the Setting Client IDs page in the ASP.NET website.

19. String.IsNullOrWhiteSpace

A small one but a good one; whereas previously a check for a null or empty string would return false if white spaces existed (you’d normally trim it beforehand to avoid this), the new IsNullOrWhiteSpace method checks for all three conditions in the one call.

20. Broad CSS standardisation improvements

ASP.NET has always tended to render some pretty heavy HTML which has been very table dependent and often not particularly accessibility compliant. The latest generation .NET attempts to tackle that by changing the default rendering of a number of controls. For the sake of not introducing breaking changes, upgraded applications are placed in a control rendering compatibility mode which ensure ASP.NET 3.5 apps continue to render identically (you can see this set in the Web.Config image further up the page). Without this, there are potential breaking changes although in my app, the extent of “breaks” was limited to a generated chart image displaying with a border as table and image controls no longer default to zero border.

A good example of where a more standards orientated approach is applied is in the RadioButtonList control. In addition to the “Table” and “Flow” layouts, we now have “OrderedList” and “UnorderedList” which output <ol> and <ul> tags respectively to surround the list items. Apparently this brings improvements for users with “Assistive Technologies” such as screen readers. More on the CSS improvements back on the ASP.NET website here.

21. Changed project templates

The standard “ASP.NET Web Application” project template (see the first image below), is now a far more comprehensive beast generating not just the Default.aspx file but also adding forms based authentication pages, JQuery script files, a default CSS file, a couple of general ASPX pages, a Global.asax and even a master page with a basic layout on it. Personally, this is of mixed use to me; I rarely use forms based authentication but I like the idea of having the master page, CSS and JQuery scripts inserted automatically.

At the other end of the scale, there is a new “Empty ASP.NET Web Application” template (second image below), which is closer in nature to the old web app template but doesn’t even give you a default page.

22. Toolbox search

Ever have trouble finding that pesky control hidden somewhere in the depths of the toolbox? With the new toolbox search feature it’s only ever a few keystrokes away. So long as the focus is on the toolbox you can begin typing the name and the control will be auto selected. You can see from the screen grab on the right I began typing “grid” and the selection has automatically moved down the GridView control.

23. Box insertion

We’ve had box selection in Visual Studio for a while; hold down the ALT key while dragging a rectangle with the mouse then cut or copy. VS2010 takes this to the next level with box insertion which provides the ability to add text to the selected box. The example on the right shows where I’ve made a zero column width selection (you can just see the turquoise selection line to the right of the text) then typed the one statement which is simultaneously inserted on each row. You really need to see this in action to appreciate it so take a look at Brittany Behren’s (Program Manager from the Visual Studio Editor Team) Box Selection and Multi-Line Editing video on the Visual Studio Blog.

24. Chart control

As with the ASP.NET Routing, this is another permanent addition first surfaced in .NET 3.5 SP1. Actually it goes back even further than that originating from Dundas and making its way into the Microsoft framework following their acquisition a couple of years back (some interesting comments on that at the bottom of Scott Guthrie’s blog entry about the new control).

In case you haven’t previously used this control, there’s quite a comprehensive suite of charts available and in most cases will save you from going out to the likes of .netCharting or Component Art and spending money on a third party tool. There’s a great sample site available which gives you the codebase for all the chart combinations so it’s very easy to get up and running.

25. Response.RedirectPermanent

The traditional approach for redirecting users to a moved page is to provide a Response.Redirect in the old page which issues an HTTP 302 to the browser (temporary redirect) and causes it to make a new request to the alternative location. Response.RedirectPermanent causes an HTTP 301 (moved permanently) which allows search engines to recognise the resource no longer exists at the originally indexed location and to instead directly reference the new location which will save the browser a round trip.

You can see based on the Fiddler capture to the right how the request for OldPage1 with the traditional redirect differs from the request to OldPage2 with the permanent redirect. The only thing I’d like to see to make this really easy to manage is the permanent redirects defined in a configuration file to save applications accumulating a raft of orphaned files that simply point to their newer counterparts.

Summary

From my perspective and based on my own needs, there’s no single “killer feature” (LINQ to SQL was this for me in .NET 3.5) rather a broad collection of enhancements that will touch many aspects of the way I build software. Looking back through my examples above, there’s unintentionally a pretty even split between .NET 4 features and VS2010 features and I think that’s pretty representative across the raft of other improvements I didn’t mention (delve into some of the links I’ve included if you want to read more).

Looking to the future, the new WPF based editor (Scott Hanselman has a good interview with Noah Richards from the Visual Studio Editor team on this), is said to provide greater extensibility for third parties to surface more features within the IDE. I’m eagerly awaiting to see how ReSharper tackles this, particularly given the number of features it previously provided that VS2010 now implements natively. Based on Ilya Ryzhenkov’s recent tweets there seems to be some pretty good things in store so fingers crossed for yet more ReSharper goodness in the near future.

I hope this has been useful for other people, I enjoyed putting it together and it’s provided me with the fastest learning curve yet for a new .NET and Visual Studio release. If it helps other people do the same it’s been a very productive days work indeed!

Creating your own custom Subversion management layer

2009-10-09T11:30:00.012+11:00

I’m a big fan of Subversion. As far as source control management products go I find it pretty intuitive and very easy for people to get to grips with regardless of their degree of technical competence. The autonomy it has from the prime time technology stacks gives it broad appeal and the fact that it’s freely available is surely a significant part of why it’s so popular today. If you’re looking for pure source control management, it’s hard to go past SVN.

Following is the process I recently went through to provision a Subversion instance with a custom management layer to surface a few key features through a web UI. I’m not going to go into great detail code wise, rather offer some info on the nuts and bolts required to couple the system together with a few key code snippets as required. Hopefully this will give someone else with a similar need a jumpstart in the future.

The goal

Recently I set out to source a new Subversion instance to support a broad range of people working across multiple, autonomous projects in different locations around the world. What I really wanted to do was find a Subversion distribution which would allow me to break rights down into three tiers:

End users; normal Subversion users who can be granted read and write permissions
Repo managers; named individuals who have the right to create a new repo on demand plus manage permissions in existing repos they have rights to
System administrators; all of the above across all repos

All the roles would need to be able to perform their duties via a web based UI and it had to run in a Windows environment leveraging Active Directory accounts for authorisation purposes. Oh, and cost is important. If I could achieve this model then the administrative burden would be moved back to the people actively creating and managing projects rather than those supporting the environment.

Now before anyone jumps up and starts singing the praises of distributed source control management systems such as Mercurial and Git, these were considered but some key constraints ruled them out. Due to various circumstances, namely product familiarity with a large audience and a desire to make the switch “low friction”, these products were off the cards for the moment.

Currently available options

There are two Subversion paths you can go down:

Run Subversion directly by installing your own instance of SVN and Apache (very good overview of that here)
Run a packaged distribution of Subversion which automates the install and provides a GUI and other easy useful management features you don’t find in the previous option

The first option is out right away as there’s no GUI. The second option is closer but out of all the packages I looked at, nothing can distribute management the way I needed them to. When it comes to creating repos and managing access rights, nothing out there seems to do the job at the moment.

If you want something done properly…

As the saying goes, sometimes you just need to do it yourself. We’re not talking about reinventing the wheel here; Subversion is a perfectly good tool in terms of managing source code, it’s really just the configuration management part I’m talking about. What I was envisaging was a web layer that would facilitate creating the repo and managing the permissions. That’s it. In my mind, the only hurdle I needed to tackle was finding a suitable API for .NET to chat to and it would be a piece of cake.

Choosing a Subversion distribution

This was actually a pretty easy choice as there’s one Subversion distribution which stands head and shoulders above the rest in terms of ease of use, price and functionality; VisualSVN Server. This is a very small footprint product (4Mb installer), gets up and running in a couple of clicks and just works every time. I’ve run it directly on a number of PCs for version control on personal projects and never had the slightest glitch with it.

The other thing VisualSVN Server gives you is a nice MMC style UI to perform common management tasks such as creating repos and managing permissions. It also facilitates server configuration activities such as the port, authentication method and whether an SSL certificate is used.

A final note on product variants; the Standard edition does pretty much everything anyone would need but there is an Enterprise edition. This adds logging functionality for operational tasks such as a user checking out code and allows for remote administration. Nice features, but not worth US$500 based on my particular requirements so I didn’t bother purchasing the license after the trial expired (hence the notice in the screen grab above).

Talking to Subversion

There are a couple of different APIs out there but the most common and most well received seems to be SharpSVN. I also later discovered that the guy behind the project, Bert Huijben, is pretty active on Stack Overflow so there’s a familiar channel available to get support on the product.

SharpSVN is essentially a wrapper for the native SVN commands and as such, it can only perform functions these commands natively expose. This is fine for activities such as creating repositories or reading log messages but the one thing it can’t do that I really needed is to manage permissions. Here’s how Bert explained it:

SharpSvn is mostly a wrapper around the Subversion api and the Subversion
api doesn't implement an api for managing subversion permissions.

and

Different configurations of Subversion have different repository permission
settings.

Right, so the only option was to look at how VisualSVN implements permissions then devise a mechanism to edit these directly.

Creating repositories

The first thing to understand with creating repositories is that the underlying command is svnadmin create, a command which needs to be run on the server. Many of the commands the SharpSVN API exposes can be executed remotely (such as reading log messages) but creating repos is not one of them.

Here’s how the new repo is created:

using (var svnRepoClient = new SvnRepositoryClient())
{
  svnRepoClient.LoadConfiguration(repoPath);
  svnRepoClient.CreateRepository(repoPath);
}

Firstly, the repoPath parameter needs to be physical path of where the repo will be created so in most cases is will be something like c:\Repositories\MyRepo. Next a bit of background on the LoadConfiguration statement:

SvnClient.LoadConfigurationDefault() (just like svn.exe) assumes the users registry hive is always available (e.g. it can read from HKEY_CURRENT_USER). It uses a few windows api calls on retrieving which directory to use for the settings.

Using .LoadConfiguration(<path>) skips these procedures as it says: Only load these settings. (+- equivalent to using --config-dir <path> on svn.exe)

In short, it’s a performance thing. As a sidenote, later on I added some functionality to read log messages from a revision range and without loading the configuration first, the process was extremely slow (actually exceeding the default ASP.NET timeout).

Managing repo permissions

As mentioned earlier, the structure of the permissions file is largely dependent on the Subversion distribution. VisualSVN Server takes the approach of using a file named authz-windows in the root of the repositories folder for any Active Directory based authorisations. This file then describes, for both the server as a whole and each repo, who has access and whether it’s read only or read and write. Here’s what it ends up looking like:

[/]
S-1-5-21-8439203732-8503583263-2345657657-123442=rw

[MyRepo:/]
S-1-5-21-2345435435-2346665755-2343245653-99804=rw

This file is granting one account rights over the entire repository and another account rights to just “MyRepo”. You’ll notice each account is referenced as a Windows SID rather than a username but you can resolve one to the other pretty easily if required:

var account = new NTAccount(username);
var sid = (SecurityIdentifier)account.Translate(typeof(SecurityIdentifier));

Creating a data layer

Something that became clear pretty early on was that there was information I wanted to capture which wasn’t natively stored anywhere within VisualSVN. For example, I wanted to collect a little bit more information about the project that was going into the repo as well as some meta data about who created it and when. I also had plans to introduce more functionality later on that would require fast access to both a list of repos and permissions and I wasn’t real keen on the idea of trying to derive these from the file system. The final straw was that managing the permissions file was going to be a bit nasty if the only source of permissions data was SIDs in a text file and this also didn’t provide a mechanism for modelling admin rights or repo creator rights.

So the data layer is actually pretty simple. All it really needs is a Repo table, a Role table and a mapping table including the identity of the user who has been granted permissions. Obviously this can be expanded a little to capture more info but there are really only two entities in the system plus the mapping table.

Securing the application layer

Now that we already have a data model to reflect the permissions we want to apply it’s pretty simple to drop a security-conscious web app on top. I just created a simple role provider implementing the GetRolesForUser method then leveraged security trimming to protect the interfaces to authorised users.

Putting it all together

Firstly the repo management page; all that’s required here is to provide a UI which allows authorised users to create new repos. Just a single textbox will suffice although in my case I ended up asking for a little more information. The role provider secures the page and the SharpSVN API creates the repo followed by a quick LINQ to SQL call to create an entry in the data layer.

Secondly the permissions management page; I redirected straight into this after creating a repo as normally the first thing you want to do is to provide access to other people. This page only needs to provide a means of selecting an identity (I had an existing source of identity data) then choosing whether to give the user admin, reader or writer permissions. After each change I’m just rewriting the Subversion authorisation file in its entirety. The only thing you need to be cautious of here is that if you then use the VisualSVN Server UI to change permissions they’ll be overwritten the next time the permissions file is regenerated from the data layer.

Making it live

Something that only struck me once I got out of development mode and onto a live server was that I was trying to expose both Apache and IIS on the one machine and that I wasn’t going to be able to run them both over port 80. My original intention had been to bind one domain name to Subversion and another to the management interface. There are a number of ways around this including writing ISAPI filters for IIS, using a reverse proxy or most elegantly, running multiple IP addresses (one for each domain) on separate NICs and binding them to the appropriate server.

All the above options were either getting clunky or would require more investment in time and probably dollars so I took the easy route and broke out the duct tape. I left port 80 bound to VisualSVN then edited the default file Apache serves up for any requests to the root (located at C:\Program Files\VisualSVN Server\htdocs\index.html) and added a JavaScript redirect to the same URL but on port 81 which I then bound to the IIS site with the management interface. To be honest, I felt a little dirty after doing it and may still pursue the multiple IP addresses option but this got things up and running quickly with the only real downside being a port appearing after navigating to the site.

Conclusion

All said and done this was a pretty simple little piece of work which got me back into some code for a while and threw a few curve balls. The underlying Subversion structure hasn’t been touched so VisualSVN upgrades shouldn’t be a problem in the future and I can still pick any repo and ship it somewhere else.

Most importantly though, the users are happy as they’re now empowered to manage their own repos without compromising on the security of others. The only problem I have now is resisting the temptation to use the platform to add more features!

Things I omitted in this post

For the sake of simplicity I’ve left out information about some of the frills I added along the way. This includes granting permissions to AD groups or to everyone, email notifications when repos are created, account the VisualSVN service runs under and some other features I added to surface repo data to the web. I also haven’t touched on pre-commit hooks; I’m saving this one for a subsequent post as it’s a project in and of itself.

Why online identities are smart career moves

2009-09-27T14:59:00.002+10:00

The final catalyst for me eventually taking the leap into the blogosphere came from an unexpected source. It was actually my own response to a Stack Overflow Question where I’d suggested that one of the best ways to make yourself more marketable as a software developer is to have an active online profile. I don’t necessarily mean to try and achieve semi celebrity status like Scott Guthrie or Joel Spolsky, rather to be able to illustrate that over time, you’ve been actively involved in the areas in which you profess to have expertise. It’s one thing to present a CV or a LinkedIn profile which says you’ve done everything from writing enterprise software to creating perpetual motion, it’s quite another to be able to reliably substantiate it.

Why is it important to me?

Let me get one thing cleared up right away; I’m not looking to change my job in the near future and for the most part, I enjoy what I do. The thing is though, building an online profile is not an overnight process and I don’t know if I’m still going to be as enamoured with my job (or my employer as enamoured with me!) in two years, five years, ten years; whatever! It takes a lot of time to build a public identity and waiting until you actually need one is just not going to work.

The LinkedIn LoveIn

There are a couple of big issues with the current system when it comes to people marketing themselves. The first is the self-ingratiating, reciprocal backslapping that goes on with LinkedIn, a practice I’m coining the “LinkedIn LoveIn”. The LinkedIn LoveIn surfaces itself through the recommendation system where person A espouses how fantastic person B is in return for an equally impressive recommendation from the recipient. Worse still, the practice is rampant between people who have been unceremoniously shuffled out of their previous employment.

This is all very warm and fuzzy for the participants involved in this practice but it doesn’t do a lot for the potential employer who later interviews them, as although the reports are all very glowing, they’re very difficult to substantiate. Of course this problem is not new, it’s just much easier to propagate now as it only takes a couple of clicks to get a recommendation from someone who can carefully construct a glowing report in their own time as opposed to the old fashioned way of being questioned on the phone and needing to provide answers off the top of their head. And to that effect, when a phone reference is done it’s usually with the subject’s superior unlike the LinkedIn model which is essentially a free for all.

And just in case you were thinking LinkedIn recommendations are really only a problem for future employers, here’s a quote from the National Law Journal in an article titled Lawyers warn employers against giving glowing reviews on LinkedIn:

Plaintiffs' lawyers, they fear, are scouring these sites, looking for evidence to dispute firings, as most LinkedIn recommendations are positive. So if a supervisor claims that an employee was let go due to performance problems but gave a rave review about him or her on LinkedIn — that, the lawyers stress, won't look so good.

Certainly makes you think twice before going out on the public record and waxing lyrical about someone’s performance. And yes, I know, the risk here is more if you’re directly involved in someone’s departure on the one had and slapping them on the back with the other but either way, you want to be pretty damn sure about who you’re recommending and what the repercussions can be further down the line.

Setting the bar very, very low

The next big issue is just how low the competence bar seems to have been set in the software industry. The number of times I’ve interviewed people and they struggled with the most fundamental of questions is staggering. Granted, recruitment agencies have a lot of blame to share but at the end of the day if you’re calling yourself a senior .NET developer and can’t even write code to declare a nullable type or instantiate a generic collection, you’ve got issues.

It’s not just syntactic ability either, it’s general awareness of the industry. I tend to ask a lot of questions about what has changed between versions of technology the interviewee professes to have expertise across or what might be in the future pipeline and very frequently I’m met with a blank stare. In many cases people just don’t seem to have an awareness of the concepts many of us take for granted. In short, there’s a lot of unconscious incompetence floating around.

The importance of an online identity

It’s very hard to consistently fake competence over a long period of time through an online identity, certainly if it involves discussion with a community of peers. That’s not to say that every word someone puts out in the public domain should demonstrate their superiority in the subject of the day, it’s simply that through their online identity a person discloses a certain amount of information about their competencies. There’s nothing wrong with .NET developer asking about how to build their first “Hello World” application in SharePoint and this sort of active information seeking is great, just don’t come looking for a job as a senior MOSS developer the next week!

Building an identity

The way I see it, you’ve got three key avenues to create an identity for yourself these days:

Twitter; probably goes without saying given it seems like every second person is tweeting these days, but Twitter is about the easiest way there is to get yourself out there.
Forums; sites like Stack Overflow are a great way to build up a public profile and of course you’re helping your fellow professional as well.
Blogging; sure, this takes a lot more work than the first two but also gives you a pulpit not limited to 140 characters or a discrete topic.

If I can’t find any information about someone whatsoever in any of these sources, it does start to make me wonder. At best the person doesn’t tend to use these medium as they’re simply passive online users, but at worst, they simply haven’t been as active in their professed subject matter as they’d like you to think. Either way, no online identity means you’re left wondering.

Recently a friend of mine told me about the interview he went through for his current job. He had difficulty answering some questions on the spot and tying them back to actual experience he’d had. He felt he’d bombed. It was much later on he found out the only reason he got the job was that his employer was able to substantiate, through his blog, that the guy really knew what he was talking about and had simply had a bad interview.

My identity

I’m pretty clear about what it is I think I do well and the general sort of thing I want to be doing in the immediate future (at least technology wise). I’ve had the better part of a dozen years actively involved in coding and while I’ve really enjoyed it (and still really enjoy it), it’s not something I do a lot of any more and quite frankly there are people out there who do it a lot better than me (refer to some of the other blog links on the right hand side). So I’m not going to focus much on actual code in this blog and the syntax I do post will probably be pretty rudimentary.

My online identity will focus on the more practical use of software within business and enabling others to deliver it effectively. I’ve found myself spending more and more time lately working to try and bridge the gap between how software developers like to work and the expectations enterprise has of them. And that’s a two way street; developers are generally not very well understood by those not actively involved in the coding process but by the same token, developers frequently have trouble relating their work back to something that makes real commercial sense.

So that’s what I’m setting out to achieve with my identity; bring some sanity to the developer / business relationship, try and show it’s something I’m actively involved in and have some idea about and all things going well, not need to rely on LinkedIn recommendations further down the track!