Diary of a Ninja

Writing your own custom ASP.Net MVC [Authorize] attributes

7/24/2011 7:59:31 AM

ASP.Net’s [Authorize] attribute is another cool feature that makes it easy to add authentication at the Controller level when building a website, but the real goldmine here is that like nearly everything else in ASP.Net MVC, you can pick apart the functionality and extend it yourself – In this post we will take a look at creating our own custom Authentication attribute.

Lately I have been involved in a number of projects that have used ASP.Net MVC as the primary web service platform because of its awesome REST based approach to controllers – WCF fans out there may be shocked to hear this, but i truly believe that building as website is a pretty solution agnostic task, and ASP.Net MVC allows us to get a lot closer to how the rest of the world thinks about “web services” in the sense that your service really is just another end point of your website just one that returns JSON or XML and therefore doesn’t really need the differentiation of being a “separate thing/entity” to your website (i.e. Web site/Web service).

ASP.Net MVC has another really neat feature tucked up its sleave when it comes to security in the form of the [Authenticate] attribute (if you have never heard of this head on over to ASP.net to take a look at a quick tutorial). The ability to have granular security baked-in at the class or method level in a controller really appeals to me, so ASP.Net MVC’s [Authenticate] attribute/annotation ticks all the right boxes by allowing you to easily hook into the ASP.Net membership provider.

The real question is; What about when you want to take an even more granular approach to the membership/[Authorize] implementation? I have seen a few developers who when facing this task start hacking the extra security into their Controller method’s code – this is not where it should be done. The primary reason for this is the fact that if you are using ASP.Net’s caching in your controllers, the ActionResult will be cached and your controller code will not even be hit!

What ASP.Net MVC Controller Action authentication looks like

The following code snippet is from the standard ASP.Net MVC project’s AccountController showing the [Authorize] attribute being used to enforce security in the user account section of the website:

[Authorize]
public ActionResult ChangePassword(ChangePasswordModel model)
{
    ...
}

The above annotation is used to let the ASP.Net MVC framework know you want to check a user is authenticated before processing the controller action – this authentication check is done even if the result of the controller action is cached (as i mentioned above this is a really important thing to point out, as if you placed your authentication inside your controller and wanted to perform caching on your output you would have to build your own caching).

Anatomy of an [Authorize] attribute

When you place the [Authorize] attribute on a Controller’s action method, a couple of calls get made to the AuthorizeAttribute class at the beginning of each request to your controller to authenticate users. There is a lot of code in the MVC framework’s AuthorizeAttribute class, but the lines containing the methods we really care about are the following (as pulled from the Microsoft AuthorizeAttribute source using .Net Reflector):

public virtual void OnAuthorization(AuthorizationContext filterContext)
{
    if (filterContext == null)
    {
        throw new ArgumentNullException("filterContext");
    }

    if (AuthorizeCore(filterContext.HttpContext))
    {
        // ** IMPORTANT **
        // Since we're performing authorization at the action level, the authorization code runs
        // after the output caching module. In the worst case this could allow an authorized user
        // to cause the page to be cached, then an unauthorized user would later be served the
        // cached page. We work around this by telling proxies not to cache the sensitive page,
        // then we hook our custom authorization code into the caching mechanism so that we have
        // the final say on whether a page should be served from the cache.

        HttpCachePolicyBase cachePolicy = filterContext.HttpContext.Response.Cache;
        cachePolicy.SetProxyMaxAge(new TimeSpan(0));
        cachePolicy.AddValidationCallback(CacheValidateHandler, null /* data */);
    }
    else
    {
        HandleUnauthorizedRequest(filterContext);
    }
}

// This method must be thread-safe since it is called by the thread-safe OnCacheAuthorization() method.
protected virtual bool AuthorizeCore(HttpContextBase httpContext)
{
    if (httpContext == null)
    {
        throw new ArgumentNullException("httpContext");
    }

    IPrincipal user = httpContext.User;
    if (!user.Identity.IsAuthenticated)
    {
        return false;
    }

    if (_usersSplit.Length > 0 && !_usersSplit.Contains(user.Identity.Name, StringComparer.OrdinalIgnoreCase))
    {
        return false;
    }

    if (_rolesSplit.Length > 0 && !_rolesSplit.Any(user.IsInRole))
    {
        return false;
    }

    return true;
}

The great thing is the AuthorizeAttribute class shown above can be inherited from and the methods above can be overridden to allow us to disable the authentication checks for the example i have for you below. These same methods can be used to add special authentication checks or anything else you desire.

An Example: A Switchable Authorize Attribute

When creating web services with ASP.Net MVC I have often been in situations where there has been a need to turn off authentication during development for access by Silverlight/Flash/JavaScript requests on the client side is pretty much a necessity to avoid drawing out the development/wasting time from liaison with other developers un-necessarily.

The example I’m going to take you through solves this in the form of a switchable [Authorization] attribute that will allow you to turn off authentication when the solution is built in DEBUG mode (i.e. during development). This will allow developers working with the site during development to test web service endpoints without having to log in every time they want to test. You could change this to run from an AppSetting property but i recommend against this as it is a pretty big vulnerability if either it doesn’t get set by developers at deployment time, or someone with nefarious intentions changes it once the site is live.

Custom Attribute class code:

[AttributeUsage(AttributeTargets.Method, AllowMultiple = false, Inherited = true)]
public class SwitchableAuthorizationAttribute : AuthorizeAttribute
{
    protected override bool AuthorizeCore(HttpContextBase httpContext)
    {
        bool disableAuthentication = false;

        #if DEBUG
        disableAuthentication = true;
        #endif

        if (disableAuthentication)
            return true;

        return base.AuthorizeCore(httpContext);
    }
}

Usage:

[SwitchableAuthorization]
public ActionResult ChangePassword()
{
    return View();
}

As you can see from the above example’s usage of the overridden AuthorizeCore() method, when built using Debug mode, the method returns true without asking the membership provider. This will allow Silverlight and Flash developers to test the site’s code from within their IDE when developing.

Adding other Attribute awareness

An even more powerful addition to the above example is the ability of the OnAuthorization() method to see the other attributes applied to the class/method at run time – allowing your attribute to change its functionality based on the other attributes applied. In my example i check that the [HttpPost] attribute is applied to the method as well by checking the count of the attribute.

Example:

public override void OnAuthorization(AuthorizationContext filterContext)
{
    //Check that the HttpPost attribute is also applied
    ActionDescriptor action = filterContext.ActionDescriptor;
    bool isAnHttpPost = action.GetCustomAttributes(typeof(HttpPostAttribute), true).Count() > 0;

    base.OnAuthorization(filterContext);
}

Where to from here?

If you take anything from the above post, it has to be yet again how awesome and powerful the ASP.Net MVC framework is. The next time you are looking to add functionality to your ASP.Net MVC site in an elegant manner, think about creating a custom attribute.

TeamCity - When 1 build agent just isn’t enough

7/26/2011 8:58:03 AM

TeamCity is one of the greatest tools to hit the Continuous Integration world, with free licensing for 20 build configurations and an easy to use interface it ticks all the right boxes (not to mention ease of use for Windows Users) – but once you splash out on an Enterprise license and grow your installation to house many build configurations you start to want more power, and this is when a second build agent is your ticket to freedom.

At my workplace all projects that we develop require compulsory continuous delivery. Our staging environments auto deploy on check-in and our live environments have only ever been deployed to by our build server not direct from a users machine (they are executed manually) – this way our team always know that deployment day is a walk in the park, and there is no stress or tension when deploying a new feature.

If you have read my posts on CI in the past, you’ll know that Continuous Integration and Continuous Delivery are both something i am very passionate in my support of.

So much so that my workplace’s love for CI and CD has meant that we have over 100 build configurations on our TeamCity server. Our team members will be working on any number of different projects at the same time, and as a result we have multiple projects deploying as part of our Continuous Delivery setup all the time and often at the same time – this can lead to a little friction, as developers can be forced to wait for their project to make it to the front of the build queue.

So what is the solution? It’s quite simple:

Moar Buildseseseses

The answer is to have multiple build agents so that you can run more than one build at a time.

JetBrains in their usual fashion have made the licensing for TeamCity simple and easy to understand when it comes to additional build agents – at the time this article was written, both FREE Professional version and the Enterprise version allow for 3 build agents included in their licenses.

This means that even with the FREE version, you can take over the world with your CI awesomeness!

After finding this out you may be a little disheartened if you run your build server on window though, as the installer for the TeamCity build agent only supports a single instance per server. With modern hardware, even the most minimal/shoved-in-a-back-cupboard/rebuilt-from-your-sisters-old-school-computer servers have more than enough resources to handle multiple build agent instances.

For non-Windows installations, running multiple build agents on a single machine is quite simple, and JetBrains lists how to do it – the Windows section of this how to relies on you not using an installer though.

There is a solution though, so “Cheer up Emo Kid” – You can install multiple TeamCity Build Agents on a single Windows installation.

Anatomy of a TeamCity Build Agent

TeamCity’s build agent is a single EXE file that runs as a service (it is located at [build agent directory]/launcher/bin/TeamCityAgentService-windows-x86-32.exe).

The build agent on start up opens a listening port as defined in a configuration file (located at [build agent directory]/conf/buildAgent.properties) The default port for this is 9090.

It then contacts it’s nominated TeamCity server (again stored in the above config file), and registers as a build agent (the default name for this agent is the hostname of the server it is installed on).

The Windows installation of TeamCity’s build agent is quite modular. Below is a screenshot of the Build Agent’s root folder:

A quick break down of the folders you’ll want to know about (some of the information below are conclusions made based on digging around, others are from information found on the interwebs):

/bin

Contains all the service control batch files (start service, stop service, install service etc etc)

/conf

Contains all the agent’s configuration files/details such as the agents name, its listening port and the address of it’s server.

/launcher

Contains all the service’s application files. The bin folder of this actually has all versions of the service not just the Windows one (Solaris, Mac etc)

/logs

Not too hard to guess…

/plugins

Contains all the Build agents plugins. This includes all the build runners and source control modules.

/temp

Contains build task temporary files (actual build run tasks – not artefacts of checked out files

/work

Contains configuration that points at the storage locations of currently running builds (in a file called directory.map).

Why do you care about all the above? You care because they unlock the ability to install more than one agent at a time!

Installing a Second Build Agent on the same server

For my example I'll assume you are doing the below as a second installation, i.e. you have a server that already has TeamCity and a single Build Agent installed (if you don’t and need help, check out my previous post on Automated Deployments with TeamCity).

On your TeamCity server Download the latest build agent installer by visiting the Agents tab of your installation and clicking the Install Build Agent link in the top right corner of the page. When you have downloaded the MS Windows Installer, run it.

When selecting what to install, stick with the default and install everything by clicking Next

When entering the installation directory, enter a New directory name. Make sure this is not the same as your initial build agent installation directory.

Click Next and wait for the build agent’s files to be installed in the directory you entered above.

When complete this will bring up a configuration window.

In this window you want to:

Enter a name for your build agent THIS MUST BE DIFFERENT FROM THE BUILD AGENT YOU ALREADY HAVE INSTALLED
Enter a port for your agent to listen on THIS MUST BE DIFFERENT FROM THE BUILD AGENT YOU ALREADY HAVE INSTALLED
Enter the server URL for your TeamCity server – this is http://localhost if you are hosting the agent on the same server as TeamCity

STOP!!!!!!

You must complete some manual changes to the configuration files of your new build agent before continuing with the installation. If you click Save button without making them, you will break both your newly installed and old build agents and have to start from scratch. You have been warned!

A note about continuing (DO NOT DO THIS YET)

The second you click the Save button, and then click Next, the installer runs a batch file to register the build agent EXE as a service – it uses a bunch of configuration settings in the file we are about to edit to complete this. What we need to do is change these configuration settings so that when the service gets registered it installs as a Second Service instead of just installing over the top of your first build agent.

Configuration Changes

Open Notepad as Administrator

Open the file [BuildAgentDirectory]\launcher\conf\wrapper.conf

Find the line that reads

# Name of the service
wrapper.ntservice.name=TCBuildAgent

And change the settings entries to something unique

# Name of the service
wrapper.ntservice.name=TCBuildAgent2

Do the same for the following entries

# Title to use when running as a console
wrapper.console.title=TeamCity Build Agent

# Display name of the service
wrapper.ntservice.displayname=TeamCity Build Agent Service

# Description of the service
wrapper.ntservice.description=TeamCity Build Agent Service

An example of the values after being altered:

# Title to use when running as a console
wrapper.console.title=TeamCity Build Agent 2

# Display name of the service
wrapper.ntservice.displayname=TeamCity Build Agent Service 2

# Description of the service
wrapper.ntservice.description=TeamCity Build Agent Service 2

Now switch back to your installation window and click Save at the settings pane.

Back in the installer window Click Next (this will install and register the service using the settings we entered above.

Now click Next again to start your new build agent

Click Finish

And you are done!

Time to go get yourself a <insert beverage of choice> to celebrate your new multiple-build-agent glory.

Where to from here?

There are a few gotcha’s that i have experienced when first moving to having multiple build agents, and they are unrelated to “hacking” this installation process but are just par-for-the-course when working with multiple build agents on a single machine. Either way, you should make yourself aware;

Multiple Builds Running Concurrently

The out of the box configuration for a TeamCity build configuration allows for unlimited instances of a build config to run concurrently – when all your agents are on a single server you probably do not want this to happen, as they probably share the same source control and may also talk to 3rd part resources such as your integration testing environment so you don’t want multiple instances of this to run at the same time.

To turn this off, simply set the maximum concurrent build value on the first page of a build configuration’s Edit Configuration Settings page to 1

Different Build Configurations Sharing The Same VCS Checkout Folder

Another problem i have run into is where you have the same solution being built for multiple environments or being deployed to multiple environments and you have both configurations sharing the same source control check out directory. This causes the builds to hang while they both try to delete/download source/build source from the same folder at the same time.

To correct this, simply make sure all your build configurations use different VCS/source control check out directories. This is changed from the Version Control Settings page of your build configuration:

Build Accessible web sites with Visual Studio – WCAG reporting

8/1/2011 7:48:19 AM

Adding Accessibility to a website for access by sight or hearing impaired users is a thought that many developers have post build. Along with this, when you’re tasked with the job of building an accessible website, Visual Studio probably wouldn’t initially be a tool that comes to mind, but Visual Studio has everyone fooled on this topic as it has this functionality covered, you just need to look a little below the surface.

Accessibility?

The World Wide Web Consortium is an international community that develops standards for the internet. A part of this entails building standards on accessibility, so that hearing and sight impaired users can enjoy the internet just as much everyone else (LOL cats included). Making your website accessible should be considered best practice, in the same way that placing wheelchair access at a public venue is, and seeing i feel the internet should be probably be more accessible that any physical venues, taking the time the implement these standards definitely helps work towards this goal.

The smart people at the W3C have come up with a great set of guidelines that are referred to as the WCAG. These checklists give developers an easy way to make sure the site they are building is accessible.

There are two progressive levels of accessibility in the WCAG checklist from the W3C, marked as Priority 1 and Priority 2. With these staged checklists is becomes easier to implement a staged roll out of accessibility to a site. Even with these checklists, when building large sites with hundreds of pages, the job of checking each item off against your encyclopaedia of pages can be hard to stomach.

This very need has lead to the creation of many different tools to help you scan your site to validate it against these lists – Visual Studio is one of these tools thanks to a nifty little feature that, though not many developers know it, allows you to check your site against the WCAG list without having to leave your favourite IDE.

Shut up and show me already…

The Visual Studio tool I show you below can be used in a couple of different ways. A lot of people use this tool on ASP.Net pages before they publish their website. I suggest against this and instead suggest you cut & paste out of the actual output of your page as there can be differences that you miss when validating against a pre-run time page.

Load the page you want to validate against the WCAG lists in your favourite web browser.

Conduct the usual View Source in your browser of your page and copy the source code to the clipboard.

Open Visual Studio and Select File > New File

Select HTML Page and click OK

Open the Source View of the new HTML page in Visual Studio

Paste the source code that you copied above in your web browser into the source code of your new HTML page

(Now comes the secret)

Select Tools > Check Accessibility to open the WCAG checker

Now select all the Priority items you want to validate the page against and click Validate

In the Error and Warning pane at the bottom of your IDE pane you can review all the WCAG Priority exceptions individually. Clicking on one of the items will take you to the exception in your page’s HTML.

(If you have moved this pane somewhere else on the screen you’ll have to refer to it at your new location instead).

Too easy!

Using Custom Web.config Transformations in MSBUILD

9/14/2011 3:02:44 AM

Web.config transformations have been around for a while now, and a lot of developers use them in their staple day-to-day environment deployment strategies – hell, Scott Hanselman was spouting about them way back in the beginning on 2010 with his “Web Deployment Made Awesome: If you’re using XCOPY, you’re doing it wrong” post. As usual though, one size does not fit all – and in the case of Continuous Integration fans out there that may have specific build-configuration-based build and deployment scenarios (such as myself), there is the need to have finer grained control over the Web.config transformation process. If this sounds like you, then this post is aimed to deliver.

What a second… what the hell are “Web.config Transforms”?

ASP.net has had a few features that that been around for what seems like forever when it comes to abstracting away or alternating between different configuration data for your website (i’m talking about configSource functionality mostly). The features were very minimal and usually created a less-than-ideal solution for developers working on big websites in multiple environments. With the advent of Visual Studio 2010 Microsoft kindly helped us all out by taking note that “hey maybe not all websites are being built for a single server with a single configuration”… Smart guys. They created web.config transformations to help deal with this problem and Jokes aside, the feature is actually pretty cool and allows you to write a base web.config file as you normally would and then transform it for each of your environments.

Your base web.config:

<?xml version="1.0"?>
<configuration>
    <appSettings>
        <add key="ExampleApplicationSetting" value="Value being replaced by Transform"/>
    </appSettings>
    <connectionStrings>
        <add name="MyConnectionString" connectionString="..." providerName="System.Data.SqlClient" />
    </connectionStrings>
    <system.web>
        <customErrors mode="Off"/>
        <compilation debug="true">
        </compilation>
    </system.web>
</configuration>

Your web.config transform (note the change to my connection string, my custom errors and my compilation mode):

<?xml version="1.0"?>
<configuration xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">
    <appSettings>
        <add key="ExampleApplicationSetting"
                 xdt:Transform="SetAttributes(value)"
                 xdt:Locator="Condition(@key='ExampleApplicationSetting')"
                 value="The new value to replace after transform"/>
    </appSettings>
    <connectionStrings>
        <add name="MySolutionDatabase" xdt:Transform="Replace" xdt:Locator="Condition(@name='MySolutionDatabase')"
                 connectionString="... My New Connection String ..." providerName="System.Data.SqlClient" />
    </connectionStrings>
    <system.web>
        <customErrors xdt:Transform="Replace" mode="RemoteOnly" />
        <compilation xdt:Transform="SetAttributes(debug)" debug="false" />
    </system.web>
</configuration>

If this is the first you’ve heard of web.config transforms and the syntax in my example above isn’t the clearest it could be, you’ll find the MSDN page on the transformation syntax a valuable read.

So what is wrong with the default usage?

“So what?” you say, “I use them every day, and they work fine”. The main problem i have with the “out of the box solution” is as follows;

The default usage creates a new config transform for each build configuration of the website they sit inside.

When using Visual Studio and you right click your web.config and Add Config Transformations you will end up with the following

This is a major problem for me, as nearly all the places i have worked that use Continuous Delivery or Continuous Integration with ASP.Net websites have used Solution Configurations for switching between different build environments, not Project Configurations.

Your website should really only have two build configurations that you use for compilation:

Debug
Release

Your different environments may be:

Local
Internal Staging
Quality Assurance
External Staging
Production
??

But all of the above environments will either be compiled in Debug mode or Release mode, so confusing build configurations with environmental configurations feels to me like you are introducing some serious “Code Smell” into your build setup.

For some environments you may want to use the same web.config transform

Another issue i have with this setup is that it assumes that you have a different transformation for each of your build configurations. What if you use the same environment configurations for a number of your build configurations? Your production-staging environment should probably be the same as your production environment for example – i don't begin to know your requirements, but i do know that mine are not as rigid as this.

For some environments you may want to use multiple tiers of web.config transformations applied over the top of each other

Another case where the default usage may not tick all your boxes is when it comes to tiering multiple configurations transformations together. Having the ability to apply transformations in functionality groups instead of environment groups is a very powerful potential usage of transformations.

So where does this leave us?

For me, this put me in the position of wanting badly to find a solution that could give me this level of granularity while still using the transformation tools built into Visual Studio and all the nice convention based functionality that comes with their usage. Up until this point i have been a big fan of abstracting my configs based on environment using something similar to below and swapping the “\Live\” string at build time:

<configuration>
    <appSettings configSource ="Configs\Live\AppSettings.config" />
</configuration>

The major problem with using this technique is that you end up risking having some configurations that are missing certain appSettings keys and a couple of other annoying niggling issues that you usually only become aware of at the time of deployment (this is a big no-no when using Continuous Integration as the main point of using it is that it removes a lot of the risks of deploying – and this very issue creates risk). So i was very eager to find a solution that uses config transforms.

“… MSBUILD is a hell of a drug…”

I am a big fan of MSBUILD when writing build scripts, if for no other reason than the fact that Microsoft’s Web Deployment Projects are basically MSBUILD scripts that integrate well within the Visual Studio IDE.

Why does this matter? Because I am about to show you a way to use MSBUILD to apply web.config transformations step-by-step, allowing you to apply them with as much fine granularity as you like…

OMFG WTG BBQ R U SERIUZZ!

Short answer: yes

The Short Version

Applying configuration transformations using MSBUILD is quite easy, all you need is the to have Visual Studio 2010 installed on the machine running your build (in my case this is usually my TeamCity server).

The bare minimum example for applying a transform is shown below (do not use this until you have read the rest of this post and become aware of “the bug”):

<UsingTask TaskName="TransformXml"
                        AssemblyFile="$(MSBuildExtensionsPath)\Microsoft\VisualStudio\v10.0\Web\Microsoft.Web.Publishing.Tasks.dll"/>
<Target Name="TransformWebConfig">
    <TransformXml Source="C:\Code\MyProject\Web.config"
                  Transform="C:\Code\MyProject\TransformFile.config"
                  Destination="C:\Code\MyProject\Web.config"
                  StackTrace="true" />
</Target>

My above example kind of explains itself, however just to clarify what this does:

Imports the assembly found at $(MSBuildExtensionsPath)\Microsoft\VisualStudio\v10.0\Web\Microsoft.Web.Publishing.Tasks.dll and gives it the task name TransformXml.
Defines a new target named TransformWebConfig (so that you can call this from somewhere else in your MSBUILD/Web deployment project file, such as in your AfterBuild section).
Inside our newly created target, we call TransformXml and pass it the following parameters
1. Our source web.config is located at C:\Code\MyProject\Web.config
2. Our transformation file is located at C:\Code\MyProject\TransformFile.config
3. The end resulting web.config file should be saved to C:\Code\MyProject\Web.config
4. Turn on StackTrace so that in my build log i can review any issues with the transformation such as rules that couldn’t be applied etc (good to have on for review).

Pretty cool eh?

With the above snippet you can apply as many different transforms as you like in your build scripts by using the command in different ways – even to put into action more advanced configurations such as applying tiered transformations.

Thar be Dragons! – Use this instead

There is a known issue with the above MSBUILD task though in that it does not close the source web.config file successfully during the application of the transformation causing it to error when reading and writing to the same web.config. This is very annoying as it causes your build to fail if you use my example above directly, because MSBUILD will try and write over the original web.config while it still has it open. To overcome this, i add a simple second step to my build task that copies the web.config to a temporary location and then deletes the file after the transform has taken place.

<UsingTask TaskName="TransformXml"
                        AssemblyFile="$(MSBuildExtensionsPath)\Microsoft\VisualStudio\v10.0\Web\Microsoft.Web.Publishing.Tasks.dll"/>
<Target Name="TransformWebConfig">
    <ItemGroup>
        <OriginalWebConfig Include="C:\Code\MyProject\Web.config"/>
        <TempWebConfig Include="C:\Code\MyProject\TempWeb.config"/>
    </ItemGroup>
    <Copy SourceFiles="@(OriginalWebConfig)" DestinationFiles="@(TempWebConfig)" />
    <TransformXml Source="C:\Code\MyProject\TempWeb.config"
                            Transform="C:\Code\MyProject\TransformFile.config"
                            Destination="C:\Code\MyProject\Web.config"
                            StackTrace="true" />
    <Delete Files="@(TempWebConfig)" />
</Target>

More advanced techniques

As i mentioned earlier in this post, i usually prefer to use a more dynamic Solution Configuration based switching in my Continuous Delivery setups for deployment, and therefore want to be able to have my transforms applied depending on the solution configuration being fired, to allow for more flexibility.

My usual website project setup includes a folder called Deployment that sits in the root of the site. In this folder i include any assets that are used as part of my deployment. This includes things like a command-line SFTP client, a web.config checker and any configuration files that are used in my build. I usually have a post build step that deletes this folder so that you don’t find things such as sftp hashes or alternate environment database connection strings finding their way onto your web host’s servers – i highly recommend you follow a similar procedure.

For my example below I base all the properties i send the TransformXml build task shown above on the build configuration parameters and the file paths being used at build time. I also refer to the transform files i have in my \Deployment\ folder shown above.

My example below, does the following:

Defines a property named WebConfigReplacement and defines it depending on the solution build configuration being used. If building a staging deployment use the staging transform file, if a production deployment use the production config transform file.
Copy the web.config file to a temporary location inside my \Deployment\ folder
Runs the config transform defined in step 1 to the temporary web.config created in step 2 and save it over the top of my web applications web.config
Delete the temporary web.config inside my \Deployment\ folder.

<!-- Setup the transformation file to use -->
<PropertyGroup Condition="'$(Configuration)|$(Platform)' == 'Deploy - Staging|AnyCPU'">
    <WebConfigReplacement>Staging</WebConfigReplacement>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)' == 'Deploy - Production|AnyCPU'">
    <WebConfigReplacement>Production</WebConfigReplacement>
</PropertyGroup>
    
<UsingTask TaskName="TransformXml"
                        AssemblyFile="$(MSBuildExtensionsPath)\Microsoft\VisualStudio\v10.0\Web\Microsoft.Web.Publishing.Tasks.dll"/>
<PropertyGroup>
    <TransformInputFile>$(OutputPath)\Deployment\Web.Temp.config</TransformInputFile>
    <TransformFile>$(OutputPath)\Deployment\Web.$(WebConfigReplacement).config</TransformFile>
    <TransformOutputFile>$(OutputPath)\Web.config</TransformOutputFile>
    <StackTraceEnabled>False</StackTraceEnabled>
</PropertyGroup>
<ItemGroup>
    <OriginalWebConfig Include="$(OutputPath)\Web.config"/>
    <TempWebConfig Include="$(OutputPath)\Deployment\Web.Temp.config"/>
</ItemGroup>
<Target Name="TransformWebConfig"
                Condition="'$(Configuration)|$(Platform)' == 'Deploy - Production|AnyCPU' Or '$(Configuration)|$(Platform)' == 'Deploy - Staging|AnyCPU'">
    <!-- Copy our web.config into a temp folder as the 'TransformXml' task has a file lock bug -->
    <Copy SourceFiles="@(OriginalWebConfig)" DestinationFiles="@(TempWebConfig)" />

    <TransformXml Source="$(TransformInputFile)"
                                Transform="$(TransformFile)"
                                Destination="$(TransformOutputFile)"
                                StackTrace="$(StackTraceEnabled)" />
    <Delete Files="@(TempWebConfig)"/>
</Target>

In closing

Hopefully this has given you food-for-thought and got your Continuous Integration juices flowing. Web.config transformations are a very cool feature, so you shouldn’t have to go without just because you are a fan of Continuous Integration – time to come play with all the other kids…

Adding filters to your ELMAH installation

9/20/2011 10:49:24 PM

One of the most common configurations people use when setting up ELMAH is email exception logging so that you get notified whenever “shit’s goin’ down” on your site. This leads to a follow on issue that stems from this in that you end up receiving 100’s of emails a couple of times a week as your website gets scanned by evil doers looking for vulnerabilities – but there is a simple and elegant solution.

I really rate ELMAH a one of the greatest contributions made to the ASP.Net community to date. Although it has way more history/lineage that its more contemporary peers such as Glimpse or the MiniProfiler, it definitely hasn’t lost any awesomeness over the years.

I have been using ELMAH for many years now and am a huge fan, although had never thought to really blog about it until reading a recent blog post from Troy Hunt on “Gootkit’s futile attack on ASafaWeb” in which Troy comes unstuck with the first problem that people using ELMAH error emails with ASP.Net MVC often find;

Below shows my Gmail account “back in the day” before i had solved the bug-bear:

I’m getting spammed by my own website…

You end up receiving hundreds of emails a week from script kiddies or botnets scanning your web site for vulnerabilities and the ensuing email storm (9 times out of 10 these requests are usually looking for the WordPress admin login page @ /wp-login.php – WordPress blogs owners take note: the evil guys are looking for you).

Drowning out the noise

There are two different ways to configure ELMAH exception filters at run time, and depending on what you want to achieve they each have their uses. The creator of ELMAH Aziz Atif has kindly documented all the ways you can implement these filter exceptions in ELMAH on the project wiki here.

Before we start, be aware that all ELMAH filtering requires that you install the ELMAH filtering module if you haven’t already done so, read on.

To do this you need to:

Add the configuration sections to your web.config

<configSections>
    <sectionGroup name="elmah">
        <section name="errorLog" requirePermission="false" type="Elmah.ErrorLogSectionHandler, Elmah"/>
        <section name="errorFilter" requirePermission="false" type="Elmah.ErrorFilterSectionHandler, Elmah"/>
    </sectionGroup>
</configSections>

Add the error filter http module to your website

IIS Classic mode version

<httpModules>
    <add name="ErrorLog" type="Elmah.ErrorLogModule, Elmah"/>
    <add name="ErrorFilter" type="Elmah.ErrorFilterModule, Elmah"/>
</httpModules>

IIS Integrated mode

<modules runAllManagedModulesForAllRequests="true">
    <add name="ErrorLog" type="Elmah.ErrorLogModule, Elmah" preCondition="managedHandler"/>
    <add name="ErrorFilter" type="Elmah.ErrorFilterModule, Elmah"/>
</modules>

Once you’ve done this you need to look at what solutions will work better for you based on what you really want to achieve, as there are a few different options:

I want to filter using web.config configuration to remove certain exceptions from my ELMAH log.
I want to filter using web.config configuration to stop receiving emails for certain exceptions.

OR
I want to filter using code to remove certain exceptions based on a complex, granular set of rules.
I want to filter using code to stop receiving emails for certain exceptions based on a complex, granular set of rules.

I believe that you should really be logging all exceptions that occur on your site as by filtering out or removing certain exceptions from your ELMAH log you are usually hiding problems in your site or disabling the ability to get a full picture of the health of your website.

If you have setup the emailing of exceptions this is a slightly different story as while you still want to log all your exceptions you do not want to get spammed every time someone scans your website for vulnerabilities – as experienced by Troy in his post. Instead you want to filter out these exceptions so that you only receive notification of issues experienced by everyday visitors to your site.

Additionally If you have setup ELMAH exception e-mails there are a number of reasons why you don’t want to be spammed by your websites ELMAH installation:

If you have spam filters installed there is a risk that over time your exceptions may start to be blocked by your spam filter, stopping you from having proper visibility over issues on your site.
If you are receiving 100’s of exception emails from your website a day there is a good risk that you will become de-sensitized to the real issues hidden within the exception jungle that has become your inbox so that when bad things actually do occur you don’t notice them – nullifying the very reason you have setup ELMAH exception emails in the first place.

Method #1 – Filtering by web.config

My favourite approach to filtering ELMAH exceptions is to use web.config additions to define the filtering rules. This allows you to add new rules without rebuilding your project and with this gives you the flexibility to make changes to your filters “on-the-go” – and as you are probably aware, over time you will come across new types of exception “SPAM”, so having the flexibility to add to these filters simply by firing up SFTP or logging into your webserver is pretty handy.

The conventions are quite flexible and can be found at the following URL:

http://code.google.com/p/elmah/wiki/ErrorFiltering#Filtering_Declaratively_via_Configuration

The Basics

A basic summary of the ELMAH web.config configuration tags are as follows

Filter Assertions

There are many different assertions that you can use to filter exceptions by condition (take a look at the full documentation for more tips), but the ones you’ll use most are shown below. The functionality includes the assertions equal, greater than, less than, regex matching, type matching, custom JScript matching, and even more awesomly - You can write your own assertions (this is crazily powerful).

<elmah>
    <security allowRemoteAccess="0" />
    <errorFilter>
        <test>
            <or>
                <!-- exception equals a filter result -->
                <equal binding="HttpStatusCode" value="404" type="Int32" />

                <!-- exception is greater than a filter result -->
                <greater  binding="HttpStatusCode" value="404" type="Int32" />

                <!-- exception is less than a filter result -->
                <lesser binding="HttpStatusCode" value="404" type="Int32" />

                <!-- exception matches a regular expression against the exception object -->
                <regex binding="BaseException.Message" pattern="The file '/[^']+' does not exist" />

                <!-- exception matches a certain type of exception -->
                <is-type binding="BaseException" type="System.IO.FileNotFoundException" />

                <!-- exception matches a custom JScript assertion that matches multiple conditions -->
                <jscript>
                    <expression>
                        <![CDATA[
              // @assembly mscorlib
              // @assembly System.Web, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a
              // @import System.IO
              // @import System.Web

              HttpStatusCode == 404
              || BaseException instanceof FileNotFoundException 
              || BaseException instanceof HttpRequestValidationException
              /* Using RegExp below (see http://msdn.microsoft.com/en-us/library/h6e2eb7w.aspx) */
              || Context.Request.UserAgent.match(/crawler/i)                      
              || Context.Request.ServerVariables['REMOTE_ADDR'] == '127.0.0.1' // IPv4 only
              ]]>
                    </expression>
                </jscript>
            </or>
        </test>
    </errorFilter>
</elmah>

Filter selectors

When you have figured out all the assertions that apply to the exceptions you want to filter out, you can join them all together using filter selectors to enable the use of more than one filter at a time (which will nearly always be the case). You can tier your filters selectors together in a parent/child relationship to even more expressively state your intentions:

<elmah>
    <security allowRemoteAccess="0" />
    <errorFilter>
        <test>
            <or>
                <!-- insert multiple filter tests for alternate assertions -->
            </or>
            <and>
                <!-- insert multiple filter tests for inclusive assertions -->
            </and>
    </test>
    </errorFilter>
</elmah>

Actually applying Filters in the web.config

Now that I've shown you the conventions used by ELMAH to express exception filters lets take a look at some examples. It is important to note that up until this point we have only been talking about writing filters to remove them from ELMAHs log, not to simply not email you about them – this is something i get to below.

Regex exception matching

Filter exceptions by Type (in this case System.Web.HttpException)

<elmah>
    <security allowRemoteAccess="0" />
    <errorFilter>
        <test>
            <or>
                <regex binding="Exception" type="System.Web.HttpException" pattern="The controller for path '/[^']+' was not found or does not implement IController." />
                <regex binding="Exception" type="System.Web.HttpException" pattern="System.Web.HttpException: A public action method '[^']+' was not found on controller '[^']+'." />
            </or>
        </test>
    </errorFilter>
</elmah>

Filter exceptions by Exception Message body:

<elmah>
    <security allowRemoteAccess="0" />
    <errorFilter>
        <test>
            <or>
                <regex binding="BaseException.Message" pattern="The file '/[^']+' does not exist" />
                <regex binding="BaseException.Message" pattern="The controller for path '/[^']+' was not found or does not implement IController." />
            </or>
        </test>
    </errorFilter>
</elmah>

Filtering Exceptions from Email only

The great part about all of these filtering assertions are they can be combined with the type of filter being used by ELMAH to filter only the Exceptions being emailed to us – which is the very point of this post.

Only specifically target errors being emailed so that we don’t receive them:

<elmah>
    <security allowRemoteAccess="0" />
    <errorFilter>
        <test>
            <or>
                <and>
                    <regex binding="BaseException.Message" pattern="The file '/[^']+' does not exist" />
                    <regex binding="FilterSourceType.Name" pattern="mail" />
                </and>
                <and>
                    <regex binding="Exception" type="System.Web.HttpException" pattern="The controller for path '/[^']+' was not found or does not implement IController." />
                    <regex binding="FilterSourceType.Name" pattern="mail" />
                </and>
            </or>
        </test>
    </errorFilter>
</elmah>

Method #2 – Filtering by code

The other approach to creating ELMAH filters is to use code, either in your own http module or inside your Global.asax.

This obviously requires more work, but also gives you a lot more of a fine grained control over your exception logging. Again, there is a solution for just plain Exception filtering to ‘not log the exception’ as well as Email Exception filtering to not send an email for a certain type of exception.

When ELMAH raises an exception, it raises the Filtering event to see if any listeners want to dismiss it. By default it looks for methods in your Global.asax that match a certain convention and executes them as listeners. There are two different method names it looks for, each for the two functionalities we talked about in the previous paragraph: Error Filtering and Email Error Filtering.

By referencing the ELMAH Binary in your Global.asax and inserting the following two methods you can then add your own logic to enable you to interact with the exception at runtime, either to extend its functionality or to dismiss the exception from ELMAH before it gets logged or emailed (depending on which method you add your logic to).

void ErrorLog_Filtering(object sender, ExceptionFilterEventArgs e)
{
    // don't log HttpRequestValidationException's
    if (e.Exception.GetBaseException() is HttpRequestValidationException)
        e.Dismiss();
}
void ErrorMail_Filtering(object sender, ExceptionFilterEventArgs e)
{
    // Don't email me about files not being found
    if (e.Exception.GetBaseException() is FileNotFoundException)
        e.Dismiss();
}

As you can see, with these code snippets you can have some pretty powerful control over what ELMAH does with your exceptions. This should conjure up a number of scenarios you may want to accomplish – Maybe you want to disable error emails when a certain condition is met (maybe an app setting), maybe you want to not log exceptions if they are a certain type. What you do is not important, but knowing the power and flexibility you have at hand can only be a good thing.

Further ideas

Using code you can plug into the Elmah exception events when they happen allowing you to really extend the functionality to get more value.

Some cool ideas i have come across:

Alter the email being sent by ELMAH to change its priority and subject depending on conditions
http://scottonwriting.net/sowblog/archive/2011/01/06/customizing-elmah-s-error-emails.aspx
Plugging into ASP.Net MVC 3 to use ELMAH as well as the ASP.net MVC exception handling
http://stackoverflow.com/questions/766610/how-to-get-elmah-to-work-with-asp-net-mvc-handleerror-attribute/779961#779961
Manually logging to ELMAH using error Signalling
http://code.google.com/p/elmah/wiki/DotNetSlackersArticle#Error_Signaling

What good Web Developers should know about sending E-mail

9/27/2011 1:29:59 AM

Nearly all websites these days send email and because of this the majority of developers assume that they “know how to send email from a website”. They continue under this assumption until they have a site or server of their get black listed by a Spam blacklist. Then they are forced to scratch their heads to try and figure out why this happened. Before you hit send on that email requesting to be removed from that Spam blacklist, let’s recap what you should be doing to make sure it doesn’t happen again.

So you’re developing a website, and you or your client wants this website to send an email.

Maybe it’s to visitors who sign up for your service, or maybe you have a “Send to a friend” service that allows your visitors to spread the word about your content or service.

Maybe you just want to send them a notification every time one of their friends virtually pokes them from your soon-to-be social network project (watch out for those Winklevoss twins).

No matter the usage, these added functionalities to your website all depend on E-mail and the assumption that these emails will get to their intended recipient.

Sadly, decades of abuse of the very E-mail services you are trying to deliver your message over by nefarious characters around the world has lead to us all being in a heightened state of Email DefCon Level 1. This means that at nearly every email gateway server around the world, system administrators and network engineers are working hard to make sure that Spam (or perceived Spam) doesn’t get through to their users’ mailboxes.

This means that you and your website need to be careful to not make the Uncle Sam’s of the network administration world go nuclear on your message.

A look at ways Spam filters can work

NOTE: Every different Spam filter on the planet does things a little differently (otherwise they’d have nothing to compete over), and I am not making blanket statements that this is how they all work. More simply, these are some of the things that Spam filters look at so that, in sharing them you as a developer can be more wary of them in your next project.

In order to make it easier to understand what I am about to explain I have going to create a virtual email for us to talk about. This example email’s basic information is:

Sender: JohnDoe@virtualcorp.com
Recipient: MarthaNoName@centuryco.com
Sending Servers IP: 202.202.202.1
Subject: Welcome to Widgets.com

Most Spam filters work under the pretence of incoming mail being given a score based on its risk of being Spam. They will do a number of checks and give your message a score – if this score is above/below the configured level for flagging your message as Spam or not it will then either let your email through to the recipients mailbox or reject it (depending on the Spam filter software being used, the response may be that your mail simply gets placed in a mailbox folder for review; think of GMAILs [Spam] folder).

Lets take a look at the lifecycle and some of the checks that occur, and then take a look at ways you can make sure your message makes the cut.

Step 1 – Your Email is received

The first thing the Spam filters usually do is take note of the domain name of the email’s sender (in our example email - virtualcorp.com) and the IP address of the sending server (in our example - 202.202.202.1) that it was received from as well as the domain name your server sends in its HELO command at time of connection.

Step 2 – Email interrogation

Now that a number of properties of your email have been noted, the Spam filter will then run a number of checks against your email. In the case of the bad guys there a usually a whole heap of things that happen outside of the list i am about to explain; such as heuristics on the email’s body content. As we are sending normal e-mail all we should worry about are the flags that could cause a false positive to occur.

DNS Background Check

The Spam filter will probably then do a reverse DNS lookup to see if the IP address of the sending server resolves to the domain name that is used when initiating connection (The HELO message) or the domain name of the email sender (in our example - virtualcorp.com).

Is this the reverse DNS hostname used in the server’s connection HELO message? Is this reverse DNS hostname included in the SPF records for the sending domains DNS? (shown later in this post). Is this server even connected to JohnDoe@virtualcorp.com?

It may then look up the Forward DNS entries for the domain name of the email’s sender (again this is virtualcorp.com) and look to see if the A record for that domain points to the IP address of the sending server or an IP in the same subnet block (i.e is the sending server actually the webserver that hosts the site or connected to this site in any way?).

All of these will be added to the email’s risk score. Some filters rate some checks higher than others.

Reverse DNS How-to

If you want to look up the reverse DNS entry for the your server’s IP address the simplest way is to visit a site like this one and enter your servers IP address.

Alternatively, open a command prompt (if using windows) or a terminal (if using a Unix flavour) and type the following:

Nslookup [IP ADDRESS YOU WANT TO LOOKUP] Hit Enter

The result when looking up the reverse DNS entry for Google’s DNS server 8.8.8.8

DNS SPF Record Check

Another common approach a Spam filter will take is to look up the SPF record and/or the Sender ID SPF record for the domain name you have as sender in your email message (again this is virtualcorp.com).

DNS SPF records define a list of servers that are allowed to send email on behalf of a domain name. They are an important way of defining servers that are not normally associated with your domain name to be recognised as being verified – if you are using a server that is shared across your webhosting ISP’s other customers, this allows you to white-list this server as being related to your website’s domain (a connection that normally might not be a clear as it should be).

Example

The following SPF record for example.com indicates that only one server can send on behalf of example.com and that is the server mail.example.com;

example.com.  TXT  "v=spf1 a:mail.example.com -all"

What does this mean for us web developers?

You may being thinking at this point:

Who cares about DNS entries and all this other bumf?

I can send email as another persons email address – I've done it before!

If you are sending email from your website, then you should care – not just because your message might not get through this once, but because a lot of Spam filters keep a score for sending servers and may either report back to a blacklist if they constantly see bad mail coming from it or may just reject all mail from you in the future. When you are sending email to a huge network of email servers (such as Gmail or Hotmail), them remembering you as sending bad mail is not a good thing.

This means that by not playing by the rules, you can actually over time get your server blacklisted.

Additionally, just because it hasn’t happened yet, doesn’t mean it won’t happen one day.

So what can be done to try and allow our E-mails get through?

After all this information, it might not be 100% clear what you can do to make sure you E-mail gets to its intended recipients and that you don’t set off any Spam false positives in the process. The list is quite small, and simple and usually if you stick to a few simple rules you should be fine.

Lets take a look at how you can use this information to your benefit.

Never send email from a sender address on a domain you do not control

If you are building a website that sends email, try to avoid sending as another user.

This means if you are building a “send-to-a-friend” form, or a “send this user an email” functionality always try and send from an email address on your domain, not as the user’s email address – you can add the user you are sending-on-behalf-of as the “reply-to” email address, but always send from a sender email address that lives on a domain you control.

If your websites domain name is “widgetsareus.com” make sure you send from an email address such as “website@widgetsareus.com” not “ladiesman69@hotmail.com”

Make sure that you have your DNS ducks are in a row

If you are sending from an email address on a domain that is hosted on another server (such as using a shared SMTP server from your webhost), get your DNS in order. If you are not in control of your site’s DNS make sure that whoever is knows about the following;

The ISP that hosts your server sets up a valid reverse DNS entry for your server’s external facing IP address that it points at a hostname that is a sub-domain of the domain name you wish to send as if possible. If you are using a shared SMTP server this is usually not possible as the server will be sending email for many different domain names not just yours – in this case SPF records are even more important.
Setup an SPF and Sender-ID SPF record for your domain name that includes the external facing IP address or hostname of your mail server. If you need some help, give the following sites a visit:
1. SPF Record Documentation
  http://www.openspf.org/SPF_Record_Syntax
2. Sender ID Setup Wizard
  http://www.microsoft.com/mscorp/safety/content/technologies/senderid/wizard/

Verify your SMTP server is configured correctly

If you control the SMTP server that is sending your mail make sure it is configured to give its reverse DNS hostname in its HELO connection greeting from its reverse DNS hostname or the domain name that you are sending from.

Also make sure that relaying is turned off – THIS IS PRETTY CRITICAL.

To check if you have this turned on, check out an open relaying tool such as the one of the following:

http://www.mxtoolbox.com/diagnostic.aspx

http://www.checkor.com/

Verify that your SMTP server is not on any blacklists

There are hundreds of Spam blacklists, but MXToolbox has a great tool for checking your servers IP address @ http://www.mxtoolbox.com/blacklists.aspx. If you find yourself on any of the above, double check that you have configured your server correctly and politely email the administrators of the blacklist and request to have your server’s IP address removed.

Summary

Sending E-mail from your website is not a complicated task (as your probably already know). Trying to assure that your email messages get to their recipients is not too hard as long as you stick to a couple of simple rules.

Hopefully the list above, will allow you to get back to doing what you do best – rocking the web.

I have written a follow up post on using DKIM and Domain Keys to “sign your email” when you want to take this a step further.

Check it out:
When you really need E-mail delivered – Signing your mail using DKIM.

Getting your Windows Phone 7 Device Syncing after upgrading to Mango on a Guest PC

10/16/2011 10:35:04 PM

During the big Mango update rush over the last 3 weeks i joined the rest of the Windows Phone 7 Development community and excitedly upgraded my phone from Windows Phone 7 Mango Beta to the real thing. I was so eager to upgrade right now that I did so on my work PC where I connect my phone as a Guest. This happily got me up and running (definite thanks to the WP7 team for doing such a great job of the upgrade experience). My troubles only began when i tried to synch my phone at home a couple of days later. Hopefully i can save a few of you the time i spent looking into this.

The first thing that greeted me upon plugging in my newly upgraded Windows Phone 7 Mango device at home was the following screen;

“…Connection Error

Can’t connect to your phone. Disconnect it, restart it, then try connecting again…”

Yay, my phone is broken, how exciting…

I can tell you that the above error message made me slightly worried. I have had my Samsung Omnia 7 for just on a year now and it has served my development journey into the Windows Phone 7 world well, however the last few months have given me some small niggling worries as the phone has started to show the signs of a phone that has been bashed on a meeting room table a couple of times a day as it followed me along in “a day in the life of Doug”. A few weeks ago it started thinking it was always critically low on battery, and this certificate issue appeared to be another fun little adventure related to a potential USB connector dyeing.

Luckily for me this was not my issue.

Instead, the above occurs when your phone changes its identifier and therefore its Zune personal security certificate is no longer valid. This issue can be caused for a million and one other reasons, but the issue itself is documented on the following MSDN article;

http://support.microsoft.com/kb/2468307

Steps to Resolve – “i don’t care, i just want my phone back”

The first way to fix this is here more for the kind folk who’ve probably landed here from a Google search. If you just wanted it fixed and don’t care how it is done, follow these steps;

Connect Windows Phone 7 phone to your computer using your USB cable.
Open the Zune software.
Revel in the awesomeness that is the connection error we are talking about above.
Click Settings, located in the upper-right corner of the screen.
Click Phone, located in the upper-left corner of the screen.
Click Sync options, located in the list at the left.
Click the Forget this phone button. This is located underneath Device options in the middle of the page.
You’re done!

Steps to Resolve – “I’m a techie and want to learn something along the way”

If however you want to solve the problem while learning a little bit about Zune’s relationship with your phone, then maybe follow this guide instead. The steps below were my initial solution, and I know that many of you out there are, like me, interested in the finer details of why this error is caused, and technically how “Forget this phone” works.

Communication between your Windows Phone 7 device and your PC is encrypted in transit over USB comms using a personal certificate that is installed on your PC by Zune. There is a new personal certificate for each Zune device you have, be it an actual Zune media device or a Windows Phone 7 phone.

The error that you are experiencing above is caused by one of the following:

Your phone’s certificate changing, maybe from an Operating System Update.
Your phone being restored from a backup firmware.
The local certificate becoming corrupted on your PC.
Using security certificate software that modifies your personal certificate store (such as provided by Spanish Government for their Electronic National Identity program). Usually if this is the case your need to uninstall the software while you follow the steps below to reset the personal certificate for your device – usually your can reinstall the software afterwards without any issue.

All of these are rectified pretty simply by Zune when you simply delete the local copy of the personal certificate for your device. This is what the “Forget this Phone” feature does, as it causes Zune to create the certificate again from your device.

Steps to “reset” the certificate that Zune uses to communicate with your Windows Phone 7 handset:

If currently running, close Zune.
Make sure you are logged into an account with Local Administrator access.
Click on Start, type certmgr.msc and press Enter to open Certificate Manager.
Once Certificate Manager has opened, open the Personal folder in the left column, and then the Certificates folder, on the right.
Now look for a certificate that reads "zune-tuner://windowsphone/...".
(if you have more than one windows phone, there will be multiple. Don’t be afraid to delete them all.
Right-click each certificate in the list that matches “zune-tuner://windowsphone/” and click Delete, and then click Yes.
Close the certificate manager
Open Zune and connect your Windows Phone 7 phone – you should be back up and synching again!

Xbox Kinect Hub Remake - An Introduction To Natural User Interfaces

10/19/2011 3:00:54 AM

I first started playing around with my Kinect at home just to get a feel for what was involved in working with the SDK – I must say that I, like many of you, was amazed, and still am, at how awesomely simple and beautifully designed the API’s in the Kinect NUI framework are. A lot of people have written little Kinect demo’s showing how to create buttons and detect hand location, but I thought I’d try something different. So I set out to mimic some of the Xbox Live's interface elements in WPF.

I must admit this post might be a little late in it’s timing – most people where blogging about the Kinect SDK at the very start of this year, and while I was part of that throng of developers excitedly crawling all over the SDK like a pack of red-cordial infected pre-school children who’ve just discovered a secret stash of unattended Space Hoppers, I simply haven't had the time I'd like to write about it. I cannot contain that silence any longer.

Game Plan

Before you set off on any little learning adventure you usually want to have a set goal in mind to make your learning experience more driven – for me this was accomplishing something different to a lot of the tutorials out there and that was to create a unified interface that closely matched the Xbox Kinect Hub that you use to navigate around your Xbox with.

You can find a walkthrough of the interface I’m talking about below.

But basically, I want something similar to the Kinect Hub (screen shots below).

Primarily I want to create an interface that has a tile based layout and allows you to “press” the buttons with your hand. This would then allow me to create applications that allow you to “navigate” around a series of pages and view “assets” – whether they be videos or images. The navigation sections won’t be covered in this post, but i plan on doing a number of other posts covering these functionalities.

I also want a “hand” cursor that has a timer based “ring” that is used to “press” the buttons similar to the screen below:

Getting started

Before we start out on our building adventure, you’ll need to grab the necessary tools for the job.

You’ll need the following:

a Kinect for Xbox 360 Sensor
Visual Studio Express of higher
.Net Framework 4.0
Kinect SDK - http://research.microsoft.com/en-us/um/redmond/projects/kinectsdk/

Once you've got all this installed and setup i recommend you do a quick restart (I've found on both my work and home pc, that there are usually issues using the SDK before you first restart).

WARNING, DON’T DO THIS AT HOME KIDS

I am a web developer by trade. I do not have much experience with WPF/Windows Forms. Therefore, any code you see in this post that you want to sip sherry and scoff at while going about your day job bringing a new accounting package to the world (I kid, I kid…) will probably be perfectly warranted and I’d love to hear from you on ways you’d do things differently or where I’ve gone wrong. Additionally, my code has been simplified to make it shorter to fit with the format of this post. This is my journey into the world of the Kinect SDK, be gentle :-)

API Design - A Work Of Art

Before we make a start on the project i have to give a quick shout out to the Kinect SDK team. They have done a stunning job on the design of the SDK and the API’s that come with it. You just have to take a look at the skeleton tracking section of the API to see how elegant the API allows your code to become:

void SkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e)
{
    SkeletonFrame skeletonSet = e.SkeletonFrame;

    SkeletonData firstPerson = (from s in skeletonSet.Skeletons
                                where s.TrackingState == SkeletonTrackingState.Tracked
                                orderby s.UserIndex descending
                                select s).FirstOrDefault();
}

Is that just beautiful or what? In my example above you get handed a list of IEnumerable<SkeltonData> and you’re off.

Follow this up with amazing little gems such as the access provided to all the major body joints of a skeleton and you start to get this sudden rush that you only get when first start hacking around with something awesome early in your career – remember the time you grok’d your first programming language and wanted to take over the world with LOGO controlled green turtles? This will bring back those glory days.

Joint rightHand = jointsCollection[JointID.HandRight];
Joint leftHand = jointsCollection[JointID.HandLeft];

On with the show

The first part of my project involved just basically setting up a new WPF project.

We are ready to get going, and the first thing i want to do is implement the Xbox Kinect Hub “hand”.

This is an interesting part of the project, as to solve it I’m going to hack a few different things together. The hand needs to have a circle complete a rotation around it when it hovers over an object. Clint Rutkas has already completed similar functionality in the Coding4Fun Kinect project with the addition of Hover Buttons that have a relatively standard eventing model, although he doesn’t use a hand as his button, and instead uses simply circular buttons. Michael Crump has a done a nice tutorial on using them here.

I want to use them slightly differently though. In both the original use in the Coding4Fun toolkit and Michael’s post they place the button’s statically in the Window and have a cursor move over the top of them to Click them. I am going to reverse this event flow and make the Button itself the cursor and fire the Hovering event inside the control whenever the button is sitting over the top of one of my buttons.

To achieve this i am going to download the Coding4Fun Kinect Toolkit, add a reference to the library Coding4Fun.Kinect.Wpf and add a hover button to my MainWindow.xaml.

At the same time I’m going to setup my MainWindow.xaml with some new properties such as a background color, bigger screen size (1680x1045 – if your screen size is smaller you may have to change this to suit), and remove the navigation bars to make it run in full screen mode.

My new MainWindow.xaml header looks like this (with the added reference to the Coding4Fun WPF and a :

<Window x:Class="KinectUserInterfaceDemo.MainWindow"
        WindowState="Maximized"
        WindowStyle="SingleBorderWindow"
        xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
        xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml" 
        Background="#1E1C37"
        xmlns:Controls="clr-namespace:Coding4Fun.Kinect.Wpf.Controls;assembly=Coding4Fun.Kinect.Wpf"
        Title="Kinect Demo" Height="1045" Width="1680">

Now that we’ve added the right references, I’m going to add my hand PNG to my project, and set it as a Resource inside my project.

With this done, i can add a Hover Button to my xaml page and set its background as my hand PNG resource above;

<Grid x:Name="theGrid">
    <Canvas Background="Transparent">
        <Controls:HoverButton Margin="0" Padding="0" x:Name="kinectButton" ImageSize="64"                                           
                                ImageSource="/Resources/Hand_Basic.png"  
                                ActiveImageSource="/Resources/Hand_Basic.png" 
                                TimeInterval="2000"  Panel.ZIndex="1000" />
    </Canvas>
</Grid>

I also want to create a bunch of “buttons” to press. So I’m going to go ahead and add the following;

<Button Canvas.Left="246" Canvas.Top="282" Height="200" Name="button1" Width="400" Background="#FFC800" 
        HorizontalContentAlignment="Right" VerticalContentAlignment="Bottom" Click="button1_Click">
    <TextBlock FontSize="36" Foreground="White" Height="64" Text="Button 1" Width="155" />
</Button>
<Button Canvas.Left="670" Canvas.Top="282" Height="200" Name="button2" Width="400" Background="#FF5A0FC8" 
        HorizontalContentAlignment="Right" VerticalContentAlignment="Bottom" Click="button2_Click">
    <TextBlock FontSize="36" Foreground="White" Height="64" Text="Button 2" Width="155" />
</Button>
<Button Canvas.Left="1095" Canvas.Top="282" Background="#6BBD46" Height="200" Name="button3" Width="400" 
        HorizontalContentAlignment="Right" VerticalContentAlignment="Bottom" Click="button3_Click">
    <TextBlock FontSize="36" Foreground="White" Height="64" Text="Button 5" Width="155" />
</Button>
<Button Canvas.Left="246" Canvas.Top="515" Background="#C80FA0" Height="200" Name="button4" Width="400" 
        HorizontalContentAlignment="Right" VerticalContentAlignment="Bottom" Click="button4_Click">
    <TextBlock FontSize="36" Foreground="White" Height="64" Text="Button 4" Width="155" />
</Button>
<Button Canvas.Left="670" Canvas.Top="515" Background="#26A0DA" Height="200" Name="button5" Width="400" 
        HorizontalContentAlignment="Right" VerticalContentAlignment="Bottom" Click="button5_Click">
    <TextBlock FontSize="36" Foreground="White" Height="64" Text="Button 5" Width="155" />
</Button>
<Button Canvas.Left="1095" Canvas.Top="515" Background="#FF0000" Height="200" Name="button6" Width="400" 
        HorizontalContentAlignment="Right" VerticalContentAlignment="Bottom" Click="button6_Click">
    <TextBlock FontSize="36" Foreground="White" Height="64" Text="Button 6" Width="155" />
</Button>

Given that the Xbox Kinect user interface usually includes a window that shows the user, I’ll also add an Image control that I’ll fill later with a single video frame that the Kinect sends us through.

<Image Name="videoImage" Height="240" Width="320" HorizontalAlignment="Right" VerticalAlignment="Bottom"></Image>

These two additions should bring it all together for us to have a nice base use interface look that I’m going for;

Now for the fun stuff. Shift to the code behind for MainWindow.xaml and follow the bouncing ball.

Inside the constructor for my xaml page, I add a new click handler to my hover button, add a new class level Runtime object and set up some handlers for it to tap into the window’s Loaded and Unloaded events.

Loaded += new RoutedEventHandler(MainWindow_Loaded);
Unloaded += new RoutedEventHandler(MainWindow_Unloaded);
kinectButton.Click += new RoutedEventHandler(kinectButton_Clicked);
_runtime.VideoFrameReady += runtime_VideoFrameReady;
_runtime.SkeletonFrameReady += SkeletonFrameReady;

As you can see from my code above, the Kinect SDK simply sets up a bunch of events that get fired for every Kinect frame that is received and processed. This means that we receive a call to my SkeletonFrameReady and runtime_VideoFrameReady methods around 20-30 times a second.

Next add some logic for the Loaded and Unloaded events defined in our constructor above. This will setup our Kinect SDK Runtime object, configure smoothing so that the joint movements we receive are slightly smoothed from the sensor (you don’t want our “hand” cursor jumping all around the screen when you wave you hand), and finally we need to setup a list of all our rectangular “buttons”;

void MainWindow_Loaded(object sender, RoutedEventArgs e)
{
    InitializeButtons();
    //Since only a color video stream is needed, RuntimeOptions.UseColor is used.
    _runtime.Initialize(RuntimeOptions.UseDepthAndPlayerIndex | Microsoft.Research.Kinect.Nui.RuntimeOptions.UseColor | RuntimeOptions.UseSkeletalTracking);
    _runtime.SkeletonEngine.TransformSmooth = true;

    //Use to transform and reduce jitter
    _runtime.SkeletonEngine.SmoothParameters = new TransformSmoothParameters
    {
        Smoothing = 0.75f,
        Correction = 0.0f,
        Prediction = 0.0f,
        JitterRadius = 0.05f,
        MaxDeviationRadius = 0.04f
    };

    //You can adjust the resolution here.
    _runtime.VideoStream.Open(ImageStreamType.Video, 2, ImageResolution.Resolution640x480, ImageType.Color);
}
void MainWindow_Unloaded(object sender, RoutedEventArgs e)
{
    _isClosing = true;
    _runtime.Uninitialize();
}
private void InitializeButtons()
{
    buttons = new List<Button>
                {
                    button1, 
                    button2, 
                    button3, 
                    button4, 
                    button5, 
                    button6
                };
}

Next, lets go ahead and add some logic into my SkeletonFrameReady method so that i can hook up the HoverButton I created above as a cursor. I’m going to be controlling its location on the screen relative to where a user’s hand is in the video frame. Then we’ll check if the hand is over any of my buttons, and if so begin the Hovering event for the Kinect button that we’ve hijacked for our cursor.

Below is the code I have for detecting my first person, finding out what laterality they are (right handed or left handed depending on which hand is higher), and then allowing them to control the HoverButton with that hand to “click” any buttons it is hovering over;

void SkeletonFrameReady(object sender, SkeletonFrameReadyEventArgs e)
{
    if (e.SkeletonFrame.Skeletons.Count() == 0) return;

    SkeletonFrame skeletonSet = e.SkeletonFrame;

    SkeletonData firstPerson = (from s in skeletonSet.Skeletons
                                where s.TrackingState == SkeletonTrackingState.Tracked
                                orderby s.UserIndex descending
                                select s).FirstOrDefault();

    if (firstPerson==null) return;

    JointsCollection joints =  firstPerson.Joints;

    Joint rightHand = joints[JointID.HandRight];
    Joint leftHand = joints[JointID.HandLeft];

    //find which hand is being used for cursor - is the user right handed or left handed?
    var joinCursorHand = (rightHand.Position.Y > leftHand.Position.Y)
                    ? rightHand
                    : leftHand;

    float posX = joinCursorHand.ScaleTo((int)SystemParameters.PrimaryScreenWidth, (int)SystemParameters.PrimaryScreenHeight).Position.X;
    float posY = joinCursorHand.ScaleTo((int)SystemParameters.PrimaryScreenWidth, (int)SystemParameters.PrimaryScreenHeight).Position.Y;

    Joint scaledCursorJoint = new Joint
                            {
                                TrackingState = JointTrackingState.Tracked,
                                Position = new Microsoft.Research.Kinect.Nui.Vector
                                             {
                                                 X = posX,
                                                 Y = posY,
                                                 Z = joinCursorHand.Position.Z
                                             }
                            };
    OnButtonLocationChanged(kinectButton, buttons, (int)scaledCursorJoint.Position.X, (int)scaledCursorJoint.Position.Y);
}
private static void OnButtonLocationChanged(HoverButton hand, List<Button> buttons, int X, int Y)
{
    if (IsButtonOverObject(hand, buttons)) hand.Hovering(); else hand.Release();

    Canvas.SetLeft(hand, X - (hand.ActualWidth / 2));
    Canvas.SetTop(hand, Y - (hand.ActualHeight / 2));
}
void kinectButton_Clicked(object sender, RoutedEventArgs e)
{
    //call the click event of the selected button
    _selectedButton.RaiseEvent(new RoutedEventArgs(ButtonBase.ClickEvent, _selectedButton));
}
public static bool IsButtonOverObject(FrameworkElement hand, List<Button> buttons)
{
    if (_isClosing || !Window.GetWindow(hand).IsActive) return false;

    // get the location of the top left of the hand and then use it to find the middle of the hand
    var handTopLeft = new Point(Canvas.GetTop(hand), Canvas.GetLeft(hand));
    _handLeft = handTopLeft.X + (hand.ActualWidth / 2);
    _handTop = handTopLeft.Y + (hand.ActualHeight / 2);

    foreach (Button target in buttons)
    {
        Point targetTopLeft = target.PointToScreen(new Point());
        if (_handTop > targetTopLeft.X
            && _handTop < targetTopLeft.X + target.ActualWidth
            && _handLeft > targetTopLeft.Y
            && _handLeft < targetTopLeft.Y + target.ActualHeight)
        {
            _selectedButton = target;
            return true;
        }
    }
    return false;
}

And for good measure, I'll add my runtime_VideoFrameReady method to fill our image with the current video frame;

void runtime_VideoFrameReady(object sender, ImageFrameReadyEventArgs e)
{
    //pull out the video frame from the eventargs and load it into our image object
    PlanarImage image = e.ImageFrame.Image;
    BitmapSource source = BitmapSource.Create(image.Width, image.Height, 96, 96,
        PixelFormats.Bgr32, null, image.Bits, image.Width * image.BytesPerPixel);
    videoImage.Source = source;
}

Last but not least, we’ll go ahead and add all the event handlers for our Buttons. My aim is just to notify you when they’ve been clicked – but you get the point so feel free to take it from here;

private void button1_Click(object sender, RoutedEventArgs e)
{
    MessageBox.Show("Button 1 Clicked");
}

private void button2_Click(object sender, RoutedEventArgs e)
{
    MessageBox.Show("Button 2 Clicked");
}

private void button3_Click(object sender, RoutedEventArgs e)
{
    MessageBox.Show("Button 3 Clicked");
}

private void button4_Click(object sender, RoutedEventArgs e)
{
    MessageBox.Show("Button 4 Clicked");
}

private void button5_Click(object sender, RoutedEventArgs e)
{
    MessageBox.Show("Button 5 Clicked");
}

private void button6_Click(object sender, RoutedEventArgs e)
{
    MessageBox.Show("Button 6 Clicked");
}

You have to agree, that was simple huh? Amazingly simple.

So there you have it – if you haven’t already, go out and have a play with the Kinect SDK. Your inner-youthful-programmer will thank you.

Here’s one I prepared earlier

I’ve taken the liberty of packaging up this project for your perusal. Feel free to download it and give it a bash!

Example project download

Your Call Is Not Important To Us

11/2/2011 6:34:14 AM

I have recently changed mobile service providers. The experiences I’ve had calling my old provider (Vodafone Australia) to cancel my plan and my new one (Telstra Australia) to buy their products have made it clear to me that companies that employ call centres for customer enquiries (mostly) have it completely wrong by modern customer service standards. Its seems really simple – so why does no one get it?

“…Your call is important to us, a customer care representative will be with you are soon as possible…”

So I've been on hold for 25 minutes to Vodafone Australia. I think they may have sensed I was unhappy and calling to cancel my account – it just seems too coincidental that the longest time i have sat on hold in memory just happens to be when i call to close my account.

So i continue to sit on hold, listening to an ever looping piano track.

But this is not a new thing. We’ve all experienced it.

Customer service Is simple stupid

If you were going to setup a company/business tomorrow you’d operate under the assumption that the customer is king. This would extend to everything you would do – good customer service is the easiest way to lift your market share; make a customers experience so good, that they sell you to their friends. I’ve heard of a number of start ups recently that have doubled their market share by doing nothing but shifting their marketing budget to their support lines.

You would take this single idea of serving the customer and spread it across all your marketing efforts and every transaction you do with them. We are in 2011 (almost 2012… wow time flies), so this would weight your decisions towards every business move you make in a big way.

If you had a twitter feed you’d make sure to reply to a mention within a few hours max (if you break this golden rule, you are simply doing social media marketing wrong). If you had an email you’d make an effort to reply to the sender within 24 hours (i believe in some instances an auto response is acceptable, but at the same time very similar to hearing on hold music – keep this in mind). If customers visited your website and needed support you’d setup a live chat. If customers called your place of business you’d pick up the phone… wait a second… what? that’s crazy.

If someone walked into your store…

This all brings me to my greatest single point of reference – human beings have been trading with each other for years, and i can bet you that when the romans were trading in their Forums they didn’t make people sit and wait for 20 minutes before talking to them. that's crazy talk; they’d lose business that way. if you have a shop front, and someone walks into your store, you engage them instantly.

This very thought is huge right now in the Australian physical marketplace. A number of banks in Australia have taken this territory as real ground to compete on. Often they actually place a concierge at the front door to help direct you to the right service representative so that you don’t have to think about your reason for being there and increase their branch dwell time – It’s a proven technique. It works. Telstra does this in their stores a lot now too.

So when did it become acceptable to treat customers like shit?

Share holders like money, CEOs like to show short term gains to get their bonuses easily. Somewhere in the last 30 years the use of Interactive Voice Response (IVR) systems came along and the rise of VOIP lead to the rise of the Indian call centre – and CEOs lipped their lips with glee.

All of a sudden the opportunity to lower costs must have seemed simply too good to miss. Couple this with a time before social media where you had to engage someone like Roy Morgan to actually call people in their homes and ask them questions to find out if they were unhappy and you have a perfect storm of customer dissatisfaction joined at the hip with a double whammy of actually having to spend more marketing budget to find out if people liked your changes or not. I don’t think many companies paid for enough market research (hell, they were trying to cut costs right? spending extra budget on research defeats the point!). The status quo changed. All of a sudden it was OK to make people listen to Bach on repeat for 30 minutes before you served them. And that’s how it’s stayed for a long time now.

Times, they are a changing…

Some companies have woken up and smelt the coffee. A number of them come to mind straight away when thinking about good customer experiences – Aami Insurance uses this very situation as it’s key differentiator by saying “You’ll always speak to a real person” in all of its marketing material. Aami is my insurance company – something must have worked.

Aami Insurance “What about Me” Campaign

If i was working at a Telco present day, assessing the shift in all my power towards phone manufacturers such as Apple and this ever pressing migration to being simply “dumb pipes” I’d be taking stock of my customer engagement options in a very serious way. I’d be looking at ways to stand out from my competitors other than download limits. Changing my call centres to move to always speaking to a real person when you call seems like an easy win… no?

End Game

While I can whinge all i want about this, I feel I may be yelling into a vacuum. If i was a service provider like Vodafone or Telstra I’d be stupid not to see the opportunity – By making sure that you speak to a real person straight away you’d be a market leader in a market where the key differentiators are disappearing fast. People do notice this kind of thing… Just sayin’.

When you really need E-mail delivered – Signing your mail using Domain Keys/DKIM

11/16/2011 1:04:17 AM

So you’ve started sending email from your site/application, your application is doing well and you are now sending more e-mail daily. With this comes the fun of finding out that most of the email you now send ends up stuck in your customers spam filters – how did this happen? You aren’t a Nigerian scammer, and people opted-in to hear from you! At this point signing your mail may be the next step.

I recently wrote a post on What good developers should know about sending mail, where i talked through the basics that all developers need to know about sending email before they simply start talking to SendMail or fire up an instance of SmtpClient.

My post covered a few things you should make sure you do when sending email through code:

Send email from a real email address (hey it may sound crazy but people might want to reply…?)
Never send mail from a domain you do not control (that’s what reply-to is there for…)
Make sure your mail server’s forward and reverse DNS records are configured properly and match their HELO/EHLO greeting.
Setup an SPF record/Sender ID DNS record for server authentication.

If you are in the category of large sender, have stumbled onto the fact that a spammer used to use your IP address for nefarious means, or simply are having issues with the delivery rates of the mail you send out, one way you can add an extra tick of approval to the mail that you send is to sign your email using Domain Keys and DKIM.

Let’s take a look at how you would do this.

Domain Keys and DKIM - A brief history

The very problem you are currently facing – your recipients being unable to validate the email you send them are from you is not a new one. Different fields on technology have faced this problem before the general development community mostly knew about it – in areas such as the Medical Industry, Defense and Engineering.

When faced with wanting a way to validate email without the hassle of using encryption and having to download the senders public PGP key, the people at Cisco developed a solution called Identified Internet Email that was aimed at allowing the sending party to verify the authenticity of a sender by investigating the senders DNS records (think encryption without the need for secrecy).

Yahoo, faced with the ever growing amount of spam on its Yahoo Mail platform loved this idea and wanted to take it a step further. This lead Yahoo Mail’s Mark Delany to develop DomainKeys to help verify the authenticity of a sender using a private key stored in the email and public key stored in a TXT record in sender’s DNS.

DomainKeys was then superseded by a combination of Domain Keys and Identified Internet Email called DKIM (Domain Key Identified Mail). Domain Keys are thought of by the people of internet land (quite literally, the Internet Engineering Task Force) to be superseded but still supported for legacy sake. Luckily DKIM uses the same key storage, and just stores it’s email header differently.

Example Domain Key and DKIM Email Header

DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed;
 d=diaryofaninja.com; h=From:To:Subject;
 q=dns/txt; s=mailserver01; t=1321412507; bh=/53LpyJkzBXKAnLlbntG4C24JqY=;
 b=Im2WP82Ra+/rdZ6Rw1KtqbvnBMULshiwLI6mgDTkPekRFNBqhHZROFimt+XhhsUD/2S6RKvzkOfjS3e5+1WBXZOnf85NPA6gnPSZbutH2va+s9/XjPFx9fHDy8Kb0wpd
DomainKey-Signature: a=rsa-sha1; c=simple; d=diaryofaninja.com;
 h=From:To:Subject; q=dns; s=mailserver01;
 b=dqE0DZWbSEUZ7ZrfRe5M5cJJlzMPgKEJdb3VTPIoHxtCrmWESR+XeFGQl5YRStfXqcc8oGE/ZbdaIsK3Ov9pBdMbVlxYvIHNXK7G4iIHE8mfms/OA/QkWwpwi9/HV/ep;

Getting it up and running on your servers

So you’ve decided to take the plunge and setup signed mail for your application.

Before you proceed you need:

Access to the DNS configuration for your domain name (we’ll be adding some new TXT records)
Either access to your mail server or the ability to change your site’s code.

Generating signing keys

The first thing you’ll need to do is generate a set of private and public encryption keys that will be used to sign your email. I am a Microsoft developer by trade so i am running on Windows – to generate DKIM keys I use Open SSL.

Download Open SSL (if you don’t already have it you’ll also need the Visual C++ 2011 Redistributable installed from here).

Open a command prompt window and run the following commands (with the domain name you are generating keys for filling in the blanks)

openssl genrsa -out rsa.private.{your domain name goes here} 768

openssl rsa -in rsa.private.{your domain name goes here} -out rsa.public.{your domain name goes here} -pubout -outform PEM

I have taken the liberty of writing a little batch file that takes your domain name as a parameter and runs these for you (an easy to use idiot proof solution).

@ECHO off
IF EXIST rsa.private.%1 GOTO keys_already_exist
 
REM Execute OpenSSL and request that it generates a 768 long private key 
openssl genrsa -out rsa.private.%1 768

REM Now Execute OpenSSL and generate a public key from the private one created above
openssl rsa -in rsa.private.%1 -out rsa.public.%1 -pubout -outform PEM
ECHO.
echo Files created for %1
ECHO.

GOTO finished_key_gen

:keys_already_exist

ECHO.
ECHO Keys for the domain name %1 have not been created because the files already exist.
ECHO Please delete your keys before trying again.
ECHO.

:finished_key_gen

Save this to a file name generate-dkim.bat and then run it by simply typing:

generate-dkim my-domain-name.com

Once you’ve run Open SSL you should have two files, a private key and a public key. The public key will go in your DNS as a TXT record and you will use the private key to sign outgoing mail using DKIM.

Adding the keys to your DNS

Both Domain Keys and DKIM use the same procedure for storing your public signing keys:

A base DNS record that lets people know whether you sign all your email or just some of it, along with a contact address for bad mail.
Any number of “selectors” or different keys. This allows you to have different keys for different mail servers, rotate keys year-to-year or any other requirement you could think of that would require more than one public key to exist at a time.

The public key you have created in Open SSL will look similar to:

Before you put this in your DNS you will need to remove the header and footer from the file (i.e. “-----BEGIN PUBLIC KEY-----“), and remove any line breaks from the string.

A Modified public key:

MHwwDQYJKoZIhvcNAQEBBQADawAwaAJhANWCczM6lXpTDUpoAbKPOuUN7fI9b0LkH9BZv/ikUrEqm6FWuBKZk2AHDH7nAVvh8pqICfVpKNSFNiZIarbDOZFL3HyDyFE1LSDTpiWQAli1+aTH7pJnvH1NR6wTaFYCfQIDAQAB

Then use this to create your DNS TXT records.

Example Domain Keys or DKIM DNS records

Base record – tells people whether all your mail is signed or not and who to send bad mail too:

_domainkey.mycompany.com IN TXT "o=~\; r=helpdesk@mycompany.com"

A “Selector” record - a domain can have multiple public keys at a time, and they are separated by different sub-domains or “selectors” – the example shows how you can split this by giving each mail server its own key:

mailserver01._domainkey.mycompany.com IN TXT "k=rsa\; p={DomainKey/DKIM public key goes here}"

Sending DomainKey/DKIM Email (through code)

Once you have setup your DNS you’ll want to send signed email – one way to do this is using code during the send process of your application (remember when you abstracted all your email functionality into a helper class? this is the time is all pays of…)

As i am a .Net developer i look for .Net solutions – there are many, many DKIM implementations in other languages if you are not a .Net kind of guy/gal, Google is your friend here.

My favourite implementation is a project called DKIM.Net on GitHub as its free, supports both DomainKeys and DKIM, and will get you up and running with minimal fuss or code-smell.

var privateKey = PrivateKeySigner.LoadFromFile(Server.MapPath("~/rsa.private.mydomain.com"));
var DnsSelector = "mailserver01";
var dkimSigner = new DkimSigner(privateKey, "mywebsite.com", DnsSelector, new string[] { "From", "To", "Subject" });
var domainkey = new DomainKeySigner(privateKey, "mywebsite.com", DnsSelector, new string[] { "From", "To", "Subject" });
        
MailMessage msg = new MailMessage();
msg.To.Add("me@mycompany.com");
msg.Subject = "DKIM test message";
msg.Body = "Hello world";
msg.From = new MailAddress("my-signed@email-address.com");
        
msg.DomainKeySign(domainkey);
msg.DkimSign(dkimSigner);

new SmtpClient().Send(msg);

If using an Open Source free library puts the fear of {INSERT NAME OF DEITY} in you, there are a number of commercial .Net products that are just as nice to use (but cost money, obviously):

Mailbee.net from AfterLogic

Mail.dll from Lesnikowski

CAVEAT:

If you are planning on using Simple rather than Relaxed DKIM signing, conducting your email signing through code can often make your signing fail as your mail server may add a new/different Date header after you send your email to it that may cause the signature of your e-mail to become invalidated (because it was signed using a different header – one with a different/earlier date).

Sending DomainKey/DKIM Email (using your mail server)

The other way you can implement the signing of outgoing mail is to employ the Domain Key/DKIM signature as the mail goes out the door – using your SMTP/mail server to do the signing.

Sadly if you are using the IIS SMTP server or Exchange to send your mail there is no out of the box support for DKIM or Domain Keys, so because of this my preferred approach is to install and configure a separate locked down (not accessible outside of the local box etc.) version of hMailserver (free) on a non-standard port. hMailServer is light weight and allows you to easily setup DKIM signing for outgoing mail with zero changes to your applications code.

A screenshot of hMailServer’s Administration console.

There are also a number of commercial software packages that allow you to use DKIM on Windows Server:

EA DomainKeys/DKIM for IIS SMTP Server and Exchange Server from AdminSystem
(a IIS SMTP/Exchange Plugin)

Smarter Mail by SmartTools
(a fully functional enterprise mail server replacement – they additionally offer a free version of their product)

PowerMTA from Port 25
(a fully featured mail server/email delivery platform)

Almost there - Testing your DomainKey/DKIM signed email

Once you have configured all of the above and want to actually test that everything is working as it should, you’ll want to know of some easy ways to test the whole thing is working:

Gmail

Gmail is by far the easiest option to testing the DKIM and Domain Key signing of your emails as they add the validation results for incoming email to the header of every email they receive. To access this simply send an email to your Gmail account, open the email and select “Show original” from the options list.

Example of the header GMAIL puts on your email:

Authentication-Results: 
spf=neutral (google.com: 10.10.10.10 is neither permitted nor denied by best guess record for domain of my-website-domain.com) 
dkim=pass header.i=@my-website-domain.com
DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed;
 d=my-website-domain.com; h=From:To:Subject;
 q=dns/txt; s=mailserver01; t=1321412507; bh=/53LpyJkzBXKAnLlbntG4C24JqY=;
 b=Im2WP82Ra+/rdZ6Rw1KtqbvnBMULshiwLI6mgDTkPekRFNBqhHZROFimt+XhhsUD/2S6RKvzkOfjS3e5+1WBXZOnf85NPA6gnPSZbutH2va+s9/XjPFx9fHDy8Kb0wpd
DomainKey-Signature: a=rsa-sha1; c=simple; d=my-website-domain.com;
 h=From:To:Subject; q=dns; s=mailserver01;
 b=dqE0DZWbSEUZ7ZrfRe5M5cJJlzMPgKEJdb3VTPIoHxtCrmWESR+XeFGQl5YRStfXqcc8oGE/ZbdaIsK3Ov9pBdMbVlxYvIHNXK7G4iIHE8mfms/OA/QkWwpwi9/HV/ep;

Port 25

Port25 offers a really cool service in the form of its email report service.

If you send an email to check-auth@verifier.port25.com they will reply with a report on the email you’ve sent them.

They take this one step further by allowing you to specify the reply email check-auth-jsmith=yourdomain.com@verifier.port25.com (example shows jsmith@yourdomain.com getting the report)

Alternative to all this additional work

If you have read this far you may be feeling that you’ve got a considerable amount of work ahead of you/behind you just to get this additional functionality out the door. You aren’t far from the truth, but if you’re even reading this post you are probably at the point where the additional work is well worth it or just plain-old “has to be done” in order to get your mail delivered to peoples inboxes.

Alternatively, If you want to have all this additional functionality without the leg work and don’t mind paying for it there is always the option of outsourcing the job of sending your application’s email to a third-party.

A number of relatively cost-effective start-ups have sprung up to serve this very need;

SendGrid (options starting at FREE for 200 emails a month) <- have used this service and they are awesome

PostageApp (options starting at $9/month for 9,000 emails)

Pros/Cons to outsourcing your email sending:

Pros

Sleep easy knowing that your email delivery problems are keeping someone else up and night.
Usually setting it up and trailing their services simply require you changing your outgoing SMTP server address (setup with no fuss).

Cons

These services often delivery email using a generic domain name ({random-Guid}@sendgrid.me) not your website’s domain name (however they usually offer upgrades to make this happen for an additional charge)
You still have to setup your DNS for their services to be “recognised” as being able to send from your domain.
Additional cost that can be FREE if you simply put in some effort.
You don’t have full control of the send end-to-end (this might make you/your sysadmin boss unhappy).

Summary

Signing your outgoing email with DKIM and Domain Keys will not save the manatees, tie your shoe laces, or make you “lose 10 pounds in just 24 hours” but it will make your email recipients (and their mail servers on their behalf) trust your email’s authenticity a lot more, and inturn help you stay out of peoples e-mail Junk/Spam folder. You may think it’s simply easier to outsource your mail sending to someone like SendGrid – either way, if you are having delivery issues and need to have your email delivered, signing your email is definitely a step in the right direction.

Scheduling TeamCity backups in Windows

12/13/2011 8:49:08 PM

Jetbrains’ build server software TeamCity is an awesome product to get up and running with continuous integration and deployment, however with its ease of operation it leaves a few nice to have business features aside. One of these is Scheduled backup – and if there is anything that your career has probably taught you to date, it’s that when things break, having a backup is priceless.

The default database storage that comes with an installation of TeamCity is HSQL and while I currently have a single build server instance presiding over more than 100 build configurations flawlessly while running HSQL for data storage there is always the fear that when something goes wrong, it'll really go wrong.

To allay this fear there are a few things that you should do:

Use a storage device other than the pre-packaged HSQL such as SQL Express (i will cover this in a later post).
Run a scheduled backup of your build server configuration regularly so that in the case of failure you can come back where you left off.

The first task ill cover in a later blog post, but the second one we’ll go through today.

TeamCity comes packaged with a number of command line tools for running maintenance tasks against your build server. The main one we’ll look at in this post is called "maintainDB" – Documented here.

maintainDB backup -C -D -L -P

The above makes it really easy flexible for use in a batch file and Windows Scheduled Task scenario – what this post will cover.

Getting stuck in

The first this we need to do is add a few environment variable paths to your Windows TeamCity server so that the TeamCity maintenance tool can find everything:

TEAMCITY_APP_DIR
TEAMCITY_DATA_PATH

To do this, login to your TeamCity Server and click on Start.

Right click on My Computer and select Properties.

Click the link on the left marked Advanced System Settings

Now click on Environmental Variables

Now click on New and enter the following paths (your configuration may be different, so please investigate these locations):

“TEAMCITY_APP_DIR” – the location of your TeamCity web application directory (on my server this is set to the default of C:\TeamCity\webapps\ROOT)

“TEAMCITY_DATA_PATH” – the location of your TeamCity data directory. This is inside the windows user profile of the user who installed it. You find this out by access the Server Configuration page in the TeamCity site/admin console (on my server this is C:\Users\dougr\.BuildServer).

Now that we’ve set the environment variables needed to make this all work, open notepad and copy the following script into a new file. You will need to change the fifth line to the reflect your TeamCity backup path.

This is a folder inside the Windows User Profile of the user who installed the server – again for me this is C:\Users\dougr\.BuildServer\backup.

@ECHO off
::
:: REPLACE THE FOLLOWING PATH WITH YOUR TEAMCITY BACKUP DIRECTORY
::
SET TeamcityDataFolder=C:\Users\dougr\.BuildServer\backup

IF "%1"=="" GOTO NoParams

SET CopyToPath=%1
ECHO.
ECHO Currently path: %~dp0
ECHO TeamCity Data Folder: %TeamcityDataFolder%

net stop "TeamCity Web Server"

CD C:\TeamCity\bin
CALL maintainDB backup -C -D -L -P -F backup-temp

net start "TeamCity Web Server"

:: GET THE LOCAL DATE IN ANY REGION
:: Taken from http://stackoverflow.com/a/3202796/115749

SETLOCAL ENABLEEXTENSIONS
if "%date%A" LSS "A" (set toks=1-3) else (set toks=2-4)
for /f "tokens=2-4 delims=(-)" %%a in ('echo:^|date') do (
for /f "tokens=%toks% delims=.-/ " %%i in ('date/t') do (
set '%%a'=%%i
set '%%b'=%%j
set '%%c'=%%k))
if %'yy'% LSS 100 set 'yy'=20%'yy'%
set Today=%'yy'%-%'mm'%-%'dd'% 
ENDLOCAL & SET v_year=%'yy'%& SET v_month=%'mm'%& SET v_day=%'dd'%


SET BackupFileName=TeamCityBackup-%V_Year%-%V_Month%-%V_Day%.zip

SET CopyToPath=%CopyToPath%\%BackupFileName%
ECHO Copying Backup to: %CopyToPath%
move %TeamcityDataFolder%\backup-temp.zip %CopyToPath%

GOTO Finished

:NoParams

ECHO.
ECHO ERROR:
ECHO You haven't defined an output path for your backup. call this script again 
ECHO followed by the Path you'd like to save your backup to. 
ECHO.
ECHO i.e. backup-teamcity.bat C:\mybackupfolder
GOTO:EOF

:Finished
cd %~dp0
ECHO Backup finished

Save this file somewhere you put the rest of your server maintenance scripts if you have any – I saved mine as C:\MaintenanceScripts\backup-teamcity.bat.

You can run this script by entering the batch file name into an elevated command prompt window and adding a parameter to indicate where you want your backup saved.

C:\MaintenanceScripts\backup-teamcity.bat C:\SaveMyBackupsInThisFolder

WARNING:
The above batch file has been tested successfully on a few of my TeamCity installations successfully. You shouldn’t have anything to fear, however if you don’t fully understand the above batch script buyer beware – do you research before using it.

What is this going to do to my server?

The batch file i have written above does the following:

Takes an incoming parameter as the destination to copy the backup to.
Stops the TeamCity Web Server service.
Runs the TeamCity maintenance tool to create a backup.
Saves the backup as TeamCityBackup-yyyy-mm-dd.zip
Copies the backup to your folder of choice.
Starts the TeamCity Web Server service.

Schedule that bad buoy…

Now we have everything in place we need to create a scheduled task to run our backup.

Open Task Scheduler and click on Create Basic Task

In the next Window give your task a name and click Next – I’ve named mine “Daily TeamCity Backup”

Now select how often you want your backup to run and click Next – I’ve set Daily as I want a regular backup.

Now set the time you want your backup to run and click Next – I’ve set mine to 12:00am so that i runs when nothing else is happening on the build server.

Now select that you want your task to Start a program and click Next

Enter the path to the batch file you saved above and enter the path you want the backup to be copied to as the only argument.

Click Next.

Tick the box marked “Open the Properties dialog for this task when I click Finish” and click Finish.

Select the radio button marked “Run whether user is logged on or not” and then click the button marked “Change User of Group” – This allows us to set what user the task runs as. As we are doing a number of things in our batch file that require Windows to be running as an elevated user we want ours to run as SYSTEM.

Enter the user SYSTEM and click ok and then OK again.

You’re done!

It’s worth testing the task once to make sure everything happens as it should by running it manually in the task scheduler settings page.

One more thing to note is that this will fill your server’s hard drive with backups every night. If you only want to keep a certain number you can use the this post in conjunction with the scripts in my post Set up scheduled log file cleaning for Windows Servers running IIS.

Kung-fu showdown – One design to rule them all

12/14/2011 6:25:34 AM

My old design lasted a long time. In fact it was the design that I launched this blog with in 2009. I loved it when I launched the site as although it wasn’t the easiest design on your reading eyes it had a certain shock/wow factor. Times change though, so i wanted a fresh look – and i wanted to claim some of that wow factor back – So if you’ve been reading for a while you’ll notice that the change has been quite dramatic. As always though, the Ninjas had to stay.

Circa 2009-2011

I realised more and more that while i liked writing content for the site i was sometimes unable to fully read an article because of a lack of readability/contrast. Part of this was my dated TFT Lenovo x61 screen (Lenovo loves to release dimly lit screens).

New look & feel

If you take a look around at the 2011 blogosphere everyone has made been making a big deal out of readability. Hanselman has made the change, Jeff Atwood of Coding Horror has always been all about it, and you only have to take a look at the new Microsoft/Windows Phone Branding to get a feel of where things are heading. Content is King and making it as consumable as possible is the key.

With this in mind, I pulled open Photoshop and went to work creating something new.

My thoughts included:

The Windows Phone Metro design guides.
Jeff Atwood’s love of white and space.
A love of more minimal designs
Making it all above the copy

Speaking of Ninjas, over the years many people have asked me how I drew or where i got my Ninjas from. You have probably seen similar ones around the interwebs a few times. Stackoverflow even has a banner that has similar Ninjas for all the world to see.

The story behind them is simple. I drew them in Adobe Photoshop using a tutorial for Adobe Illustrator from Chris Spooner over at Spoon Graphics. I must thank Chris for this great tutorial – even though it actually taught me more about Photoshop than it did Illustrator as i had to do it the hard way and create vectors and smart objects in Photoshop.

HTMLification

Everyone needs at least one Dutch mate…

I am very lucky to be surrounded by oober cool/intelligent developers at my day job, so was very excited to have our resident Dutch HTML/Client-side Guru John Van Der Loo around to help me with making the new design awesomely HTMLified. Like most developers I cut my teeth on HTML years ago, and like most developers who don’t specialise in front end development you’ll know that it can sometimes be a bit of a pain in the ass when it comes to niggling little cross browser issues. That’s why i brought in my resident expert.

I wanted it not only to be beautiful, readable and easy on the eyes, but i also wanted maximum cross browser friendliness with any bells and whistles that come with it along the way. John is one of those guys when it comes to perfection, so there was no one better to have around when putting it together.

One size does not fit all

One of the great things about using new technologies such as HTML 5 and CSS 3 is that you get media-query support that allows you to build pages for multiple screen sizes. No longer was the site going to be limited to 1024x768 viewers (or me on my Lenovo x61). Everyone could be happy. Even those of you tuning in on your mobile devices on the train to work (well at least if noone else is, i am…) get get involved. If you have a large screen you can see it all happen before your eyes magically just by resizing the very window you are reading this post in.

Mobile

Mobile landscape

800x600

1024x768

1440x900 and higher

HTML 5/CSS3 Love

Another really cool feature of CSS 3 is transforms. If you hover over the Ninja’s in a browser that supports transforms you will see an Easter egg hidden on the site – the Shuriken’s spin on the second Ninja.

@-webkit-keyframes rotate {
    from {     -webkit-transform:rotate(0); }
    to {    -webkit-transform:rotate(360deg); }
}

@-moz-keyframes rotate {
    from {     -moz-transform:rotate(0); }
    to {    -moz-transform:rotate(360deg); }
}

@-webkit-keyframes rotate2 {
    from {    -webkit-transform:rotate(720deg); }
    to {    -webkit-transform:rotate(-720deg); }
}

@-moz-keyframes rotate2 {
    from {    -moz-transform:rotate(720deg); }
    to {    -moz-transform:rotate(-720deg); }
}

@-webkit-keyframes rotate3 {
    from {    -webkit-transform:rotate(-1080deg); }
    to {    -webkit-transform:rotate(1080deg); }
}

@-moz-keyframes rotate3 {
    from{    -moz-transform:rotate(-1080deg); }
    to {    -moz-transform:rotate(1080deg); }
}

John has also used all the new HTML 5 semantic tags such as article, nav, header, footer and section (along with using Modernizer to allow all those IE 6/7 users to still see everything). Awesome!

All up I really like the new look and I hope you do too.

Let me know what you think.

MCTS 70-515 Web Applications Development with .NET 4 - In Review

1/1/2012 2:12:01 AM

Recently I studied and sat a Microsoft Certification for ASP.Net 4.0 and MVC. As much as some people on the web cause grief to anyone who’s interested in certifications I must state that I actually did learn a lot about other parts of the framework that day-to-day I never touch. The great thing about this experience was that while yes I did learnt about some parts of the ASP.Net framework that I’ll never touch because in the real world you wouldn’t use them, I also found I learnt about a number of other parts of the framework that I didn’t know were there and these new bits of knowledge will help me daily.

Web Applications Development with Microsoft .NET Framework 4

A year and a half ago I started studying for the certification 70-562 because my old employer needed a few more developers to get certified to maintain their Microsoft Partnership level (as some members of staff that were certified had left). The 70-562 exam covers ASP.Net 3.5 from a more classic approach, learning about how to use lots of web controls to do everything, and at the time it felt a little dated as nearly everyone i knew working with ASP.Net was from a “I can code HTML blindfold and prefer having control over things” background and therefore wrote their own HTML without the need to use controls to create mangled HTML or use tables for things on their behalf (I kid… I kid…).

The 70-515 exam feels a lot more grounded in the real world that we all live and work in though, as it covers all the ASP.Net 3.5 controls etc. with a little less importance placed on them and also includes ASP.Net MVC 3 as well – which seeing my work mostly consists of only ASP.Net MVC this addition made me feel like I could really get value from the training and study that comes with doing this certification.

Microsoft recommends that developers wanting to sit this exam have the following experience:

Candidates for this exam are professional Web developers who use Microsoft Visual Studio. Candidates should have a minimum of two to three years of experience developing Web-based applications by using Visual Studio and Microsoft ASP.NET. Candidates should be experienced users of Visual Studio 2008 and later releases and should have a fundamental knowledge of the .NET Framework 4 programming languages (C# or Microsoft Visual Basic). In addition, candidates should understand how to use the new features of Visual Studio 2010 and the .NET Framework 4.
Candidates should also have a minimum of one year of experience with the following:

Accessing data by using Microsoft ADO.NET and LINQ
Creating and consuming Web and Windows Communication Foundation (WCF) services
State management
ASP.NET configuration
Debugging and deployment
Application and page life-cycle management
Security aspects such as authentication and authorization
Client-side scripting languages
Internet Information Server (IIS)
ASP.NET MVC

Areas covered

Configuring and Deploying Web Applications (10 %)
Consuming and Creating Server Controls (20 %)
Working with Data and Services (17 %)
Troubleshooting and Debugging Web Applications (16 %)
Working with ASP.NET AJAX and Client-Side Scripting (15 %)
Targeting Mobile Devices (5 %)
Programming Web Applications (17 %)

From the above topics about the only thing I think I have no use for and will probably die without ever putting the skills I've learnt into practice are the sections on SharePoint Web Controls. I won’t go into why I feel this is the case as there are enough people on the interwebs bitching about Sharepoint without me adding my 2c.

Study Material

I spent about 3.5 months studying for the exam and based my study around the following routine:

Studied and took notes from one chapter of the 70-515 Microsoft Training book a week including doing all the example tutorials and quick-quizzes.
Used study-mode on the Microsoft training book software (useless and full of wrong answers).
Used study-mode and certification mode on the MeasureUp training software for 70-515.

I must share my opinion on a few things;

The testing software that came with the training book had around 30% wrong answers for the questions on it. That’s right 30%! It also only has a total of 55 questions on the CD – to put this in perspective my actual exam was 76 questions long, so I didn’t feel the software could possible cover enough areas to gain much value from them. I almost recommend that anyone purchasing the training book literally throw the testing software CD in the bin. It’s that bad. The software is also quite buggy as it crashed on me a few times.

Why care about Testing Software?

A lot of people recommend against doing any training example questions as part of your study and prefer that you just “know the material like the back of your hand”. I disagree with this, as I think it is quite important to be familiar with the format that you sit the exam under as it simplifies the process of you sitting it by bringing it down to just having to know the correct answers instead of having to take in the exam process on the day as well. I also think that if you’ve never sat a Microsoft or Cisco etc exam before that it is important to familiarise yourself with the style of questions as they nearly all are asked in a sneaky way to usually test you on more than just common knowledge (i.e. showing you 3 correct answers but only one truly correct one based on the context of the question).

When I started to use the trial exam software included in the Microsoft training book I instantly felt I was wasting my time. A large number of the questions on the CD either had small mistakes in their code samples that would mean that in the real world they wouldn’t compile, or had blatantly incorrect answers.

This made me want to find a more reliable source for training questions.

The options I looked at where:

Measure Up 70-515 – 150 Questions – $109.00

Transcender 70-515 – 114 Questions – $139.00

They both have trial software that allows you to get a feel for their products. The Transcender software definitely feels more thorough as a software product (the MeasureUp software feels like it was written before the end of the cold war – maybe an over exaggeration but it definitely feels dated) however for the price versus number of questions I took the MeasureUp software as i felt it would probably give me a greater variety of questions and therefore I’d get more value out of it.

In the end there were actually around 5 questions in the MeasureUp software that had similar issues to the Microsoft training software in that there were small errors or code that wouldn’t compile. Other than this though I did feel that the MeasureUp software did help me greatly in refining my speed in being able to deduce what was really being asked in the questions, because as previously mentioned, the Microsoft style of asking questions is often quite deceiving – MeasureUp’s software replicated this quite well.

I must also add that on emailing MeasureUp informing them of the questions that had issues, they refunded my entire purchase price and released a new version of the question pack (the software checks for updates every time you load it) within a short time frame – this is quite commendable as it shows that they both care about the quality of the questions and also care about their customers.

How I found the certification test.

The test that i sat on the day was 76 questions over 3 hours. There were a number of disclaimers I agreed to before sitting the test and one of these was that some of the questions where un marked to test upcoming questions for future exam participants and because of this I am unsure how many where actually scored. All the trial exam software I sat always talked about the test being 54 questions, so the answer to how many questions were actually tested lies probably somewhere in the middle.

Overall I found the exam quite balanced to the split that I showed it to be testing above in this post, however I found it interesting that a number of the MeasureUp trial exam software questions where actually in the real test which added a little relief. Apparently the exam questions are randomly pulled from a large question pool for each participant, which makes me think that if this is the case that i must have had a lot of luck getting questions i had already studied.

I did find the test to definitely not be a walk in the park. I consider my knowledge of the ASP.net framework to be quite high and I was definitely slowed in my answering of a number of the questions. Because of this I recommend studying the material very thoroughly if you have not being doing ASP.net for as long as they recommend.

My moment of anxiety

On the day of my exam I sat my test at a Prometric testing centre on Market street in Sydney. Upon completing the test it congratulated me on passing and then asked me to wait while my score was submitted. Halfway through this wait, the testing software crashed and a Modal window told me to speak to a Prometric customer support member. This completely freaked me out as I instantly worried that my score wouldn’t be recorded and I would have to sit the test again. I was told all was fine however they were unable to give me a score print out and that Microsoft would contact me once the test result had been processed which could take up to a week. When my colleague who’d sat the test on the same day received not only his print out on the day but also was contacted by Microsoft the next day I definitely had a day or two of rising fear of being “lost in the system” – happily I have since received my notification from Microsoft, but the wait definitely didn’t make my day :-)

Conclusion

Overall I found the certification a good test of broad skill and knowledge in the ASP.Net framework and would recommend it to other developers who want to not only add another piece of paper to their resume but also because i feel I learnt a number of things i wouldn’t have learnt through my day-to-day job that i can both put to use and also add to my confidence in knowledge of ASP.Net.

Corporate accounts payable Nina Speaking. Just a Moment… Let’s talk about Workplace noise

1/10/2012 6:16:07 AM

Whether you’re working on a website for your local corner store, or writing software on the next super-jumbo jet’s critical systems, distractions can contribute to a pretty unhappy working life as a developer or technical writer. Our very trade requires making calculations, creating complex logical paths and holding large amounts of information in your mind all at the same time. Please give us some peace and quiet while we do this… Please!

In the Mike Judge movie Office Space they show a scene where the main character Peter Gibbons (Ron Livingston) arrives to work and slowly over the course of a morning experiences the living hell that can be a programming job in a loud cube farm; including being seated next to a department secretary who repeats her greeting ad nauseam (this very scene is where I got the title for this post).

Developers want quiet workplaces because there are less distractions. Less distractions means more efficiently written, elegant code (yes this is an assumption without stats to back it up, but I can tell you from my own personal experience that the best code I write is either at night after everyone at work has left, in the morning before everyone has arrived or quietly in my home office after a long day, as all these places offer less distractions).

Studies have repeatedly shown that distractions reduce performance in the workplace. And yet these studies aren’t actually specifically looking at developers at all! Most of the studies I've read simply look at cognitive performance and recall – no wonder as developers we’re all a grumpy non-communicative bunch with our headphones in.

And yet at nearly every place I’ve worked that isn’t a tech start-up, management has implemented or tried to implement open plan workspaces “for easy interaction”. They then usually add workplace music or play the music channel from cable TV under the guise that it will make the workplace “more fun” and “more productive”.

Are they all crazy?

He’s plugged in!

In the movie The Social Network, whenever someone tries to talk to a developer working with headphones on, Sean Parker (played by Justin Timberlake) discourages disturbing the them by yelling “He’s plugged in!”. I have had some non-developer co-workers mention this in the past as a reason for ignoring the problem of a noisy workplace, but this is tantamount to an Ostrich placing its head in the sand. “But he can just listen to music on his/her headphones to help him/her concentrate if he/she has to”. Some workplaces actually have the complete opposite and ban headphones as they may be considered anti-social…!!

Just to show you how much of a large increase in productivity that workers can gain from not having intermittent distractive noises in the workplace, Herman Miller did a study of devices that mask the sound of other things in the workplace by creating white noise to drown out everyone around you. The results of this study are astounding to say the least:

Studies of sound masking systems have reported productivity gains of 8 to 38%, stress reduction of up to 27%, and job satisfaction increases of 125 to 174%.

Stepping back from these results would lead you to believe that placing these devices in every workplace would be the thing to do right away – hell, think of the ROI within 2 years of installing them!

Anyone who works in sound or music production will tell you though that this is a fallacy, and that if you have one loud channel in your mix it’s always better to turn the loud channel down than the soft channels louder. Add to this that I can only assume that after sitting inside a workplace that sounded like a datacentre server room all day (to drown out all the noise) I may have been more productive but i almost certainly won’t have the greatest hearing after a few years.

We are legion hear us…

While some of my non-developer readers may think that all of this may sound a bit excessive and immature (“we can’t all have what we want, people making noise are just doing their job”), I must remind most of you that if you are in an industry that relies on software products or services workplace satisfaction is really what most employee longevity boils down to. Good developers know this and will simply move on if you don’t create the right environment for them to go about their business – again this may sound immature or unrealistic but you only have to conduct your business badly for a period of a few months to a year before you’ll start to experience what we in the biz call the Dead Sea Effect.

Basically, this boils down to a simple truth: If you make your employee’s working days filled with multiple, never read again TPS reports and constant aural distractions, your good developers will leave and you’ll be left with only the average ones who are too scared of losing their jobs to make a peep. This is not the best outcome for any business that relies on software production – usually losing even a bad developer costs you dearly as you train a new one up on your businesses processes.

When it comes to loud noise in the workplace I am not the first person to bring a mention of this over time – pretty much anyone worth their salt in the programming game has talked about it at some time.

As usual Jeff Atwood eloquently and humorously makes his point very well in his post in 2004 This is your Anti-Productivity Pod.

Joel Spolsky does an equally good job when talking about The Joel Test for Software Development Companies, by placing quiet on a list of must haves for anyone writing software as part of their job.

So what do we do about it

None of the answers below may be rocket science, as nearly all workers enjoy a bit of peace of quiet even if they aren’t in the role of software production, but some of the things required to make it happen can sometimes appear to be simply too hard for some management teams to even begin to think about.

Create a workplace that is quiet.

This has got to be first and foremost. If you have a loud office environment maybe move your developers to a separate part of the building or floor, you may think we’ll feel like we are missing out by not hearing about how many times Barry from finance barfed at the last work party – but I can almost assure you we won’t; we’re developers after all, so a lot of the time hearing about how many times the guy next to you totally p0wned on Battlefield 3 last night or how sweet the surf was before work today is scales of economy more exciting or fulfilling to us.

Create a culture of respecting each other’s aural and brain space.

As with most things that add to the vibe of an office culture, culture in general is something that is built, it doesn’t happen instantly or magically; so start small today.

While you will never hear me mention The Social Network and “there is some good insight there” in the same sentence ever again, if you take the movie at face value I think they’re onto a good thing here – if Sean Parker really did yell at people for disturbing programmers he has moved up a notch from the douche of the universe that the mass-media has painted him as over the last decade.

If you see that someone is busy in thought smashing away on a keyboard, maybe walking up to their desk to ask them for the 10th time today whether they are any closer to fixing “that bug” will be counterproductive – maybe send them an email instead. If you use internal chat or messaging and someone is marked as “away” or “busy” maybe they actually are – again maybe shoot them an email or wait for their status to change.

If you see someone else doing something similar to another colleague maybe stop them and have a word – when you are next busy and someone does the same thing you’ll thank yourself. Together it’s never too late to implement change.

Conduct meetings in meeting rooms or separate workspaces away from other workers.

Last but not least, if you think that general banter and office music are a bad distraction, placing 10 people in an open plan meeting room in the middle of an office is taking things to another level. Organise meetings away from workspaces and people’s desks. If you have segregated areas that are away from the main office floor such as meeting rooms or glassed-in sections of the office floor, use them. Your developers and co-workers will thank you.

We can all create a little more Zen in the our workplace, but recognizing that it’s actually a problem is definitely the first step.

Arachnophobia – Spider, Spider go away, come again another day

1/18/2012 9:46:27 PM

While working on soon-to-be-released projects there has often been a need to make a staging/testing website publicly accessible for client testing. This is a slippery slope if search engine spiders get in and index your site before the rest of the world is meant to see it (it happens more than you’d like to think) – If it happens to be a website you are building for something that the rest of the world shouldn’t see yet such as a product/service launch, having it leak too early can often make or break you. They have a word that describes this very fear of spiders – it’s called Arachnophobia.

Arachnophobia is a library designed to address this very concern. It contains a .Net HttpModule and HttpHandler that tells visiting search engine spiders that you don’t want your site indexed.

But I didn’t link to my site anywhere?

When you really need search engines to index you they’re nowhere to be found – other times they poke their nose in when you really wish they wouldn’t.

One of the sometimes strange problems that developers come unstuck with when it comes to being indexed too early is that they assume that because they haven’t linked to their site anywhere that search engines won’t index your site.

The main reason they face this issue is that sites that list registration and de-registration of domain names often link to new or changed domains. This means that if Google comes along to one of these sites and finds a link to your new domain name www.super-cool-unicorns.com it will come for a visit without a second thought.

What is Arachnophobia?

Arachnophobia is two things rolled into one. I have tried to make installation a frictionless exercise so that you can get onto more important things in your day.

An ASP.Net HttpHandler that implements a robots.txt file with a disallow all telling spiders to stay away.

User-agent: *
Disallow: /

This means that when a search engine spider comes to visit your site, if they aren’t evil spiders and abide by the “non-standardised” standard for robots.txt (there is no actual standards body for this, as it was simply agreed in June 1994 by members of the robots mailing list), they will attempt to download the file /robots.txt from the root of you

An ASP.Net HttpModule that adds the Google/Yahoo X-Robots-Tag header to every asset/page request.

HTTP/1.1 200 OK
Date: Tue, 25 May 2010 21:42:43 GMT
(…)
X-Robots-Tag: noindex
(…)

What it is not

Let’s be clear – this is a beta project so if it eats your babies, I take no responsibility. This carries the “works on my machine” badge, so Buyer beware…

Arachnophobia doesn’t actually stop spiders from indexing your site – your site will still be accessible in a web browser for any passer-by to see. Relying on this “security through obscurity” approach should not be considered best practice if you want to hide your site.

What Arachnophobia does do however, is easily setup a robots.txt file for your site and also add an http header to each page request telling spiders “please don’t index my site” – this is a passive approach though and will only work for the main search engines such as Yahoo, Bing, Google etc. There are still plenty of bad web spiders crawling the web that disobey this form of “request” to not index.

For mission critical web applications that you want to show to a client you should at the very least setup your webserver to only serve requests to a white list of IP addresses (i.e. you and your client’s IP addresses) or hide the site behind some form of authentication.

Why do you care?

You may be left thinking that you could easily add a robots.txt and the specific header to IIS manually easily, so what is the point?

The idea behind Arachnophobia is threefold:

When you need to add robots.txt and http headers nearly instantly using Nuget (and then take them away just as easily).
When you want to apply these features to a whole server at once (the binary is signed and can be added to the GAC and i have a setup project in the works to make this even easier).
In future i will be adding more features that will extend this functionality (such as hard blocking based on user agents etc).

Where can I get it

Add it to your ASP.Net website using the NuGet package manager:

Or download from NuGet here.

Download the Source Code from GitHub

Download the Binaries from GitHub

Installation

If you download the Binaries and want to install it yourself manually you simply need to follow the bouncing ball:

1. Reference the “Arachnophobia.IIS.dll” Binary in your ASP.net website

2. Add the Http Handler and Http Module sections to your web.config

<system.web>
    <httpModules>
        <add name="AracnophobiaHeaderModule" type="Arachnophobia.IIS.RobotHeaderModule" />
    </httpModules>
    <httpHandlers>
        <add path="robots.txt" verb="*" type="Arachnophobia.IIS.RobotsTxtHandler" />
    </httpHandlers>
</system.web>
<system.webServer>
    <validation validateIntegratedModeConfiguration="false" />
    <modules runAllManagedModulesForAllRequests="true">
        <add name="AracnophobiaHeaderModule" type="Arachnophobia.IIS.RobotHeaderModule" preCondition="integratedMode" />
    </modules>
    <handlers>
        <add name="AracnophobiaRobotsTxt" path="robots.txt" verb="*" type="Arachnophobia.IIS.RobotsTxtHandler" preCondition="integratedMode" />
    </handlers>
</system.webServer>

8 Must-Have Tools for Windows Phone 7 Development

1/23/2012 5:33:41 AM

After developing Windows Phone 7 applications in my spare time over the last year, I've collected an assortment of tools that make developing apps so much easier than when I first jumped in to Silverlight/Windows Phone 7 development. I only wish I'd know about them all when I first started down the Windows Phone 7 path. Whether you are just starting out or have been developing Windows Phone 7 apps for a while now, there’s something here for everyone.

Silverlight Spy

This is simply a must have for Windows Phone 7 development. Silverlight Spy is many things, but the easiest way of thinking about it is as a Firebug replacement for Silverlight/Windows Phone 7.

With Silverlight Spy you can run your Windows Phone 7 application and make design/layout changes in real-time and see those changes before your very eyes. Combine this with the Metro Grid Helper class from Jeff Wilcox so that you can easily visualise the Metro design guidelines and make changes on the fly and you have an amazing combination!

Unlike Expression Blend (and more like Firebug), you can simply select an element in your control tree and make those “just a little bit to the left” tweaks. You’ll note from my example image on the right of my Google Analytics app InTheKnow that it even highlights the control you have selected on the emulator screen.

It also have a built in Isolated Storage viewer, Screen shot tool, Resource viewer and XAP package explorer.

The licensing for Silverlight Spy has two options, a FREE limited version that doesn’t support Windows Phone 7, and a personal license for 69 euro.

http://firstfloorsoftware.com/silverlightspy/

In a word – Priceless

I will do a more detailed post on Silverlight Spy soon.

Isolated Storage Explorer

The Isolated Storage Explorer project over on CodePlex is an amazing freebie extension for your Windows Phone 7 development ‘getup’.

In very basic terms it allows you to copy the Isolated Storage for your Windows Phone 7 application to a local folder so that you can view/edit etc. the files that are stored on your device – you can then copy these files back onto your device allowing you to easily upload sample data to test certain situations that can be a bit harder such as large asset/image libraries, corrupted/invalid serialized data and performance issues from opening/saving overly large files.

This tool has helped me troubleshoot a number of weird serialization issues I've had in the past, and with the addition of Background Agents in Windows Phone 7.1 SDK its been a priceless tool for helping troubleshoot/view activity logs on my device during testing.

http://istool.codeplex.com/

Silverlight Toolkit

The Silverlight Toolkit is a package of controls that extend the Windows Phone 7 SDK and released under the Microsoft Public License.

It is filled with a number of controls that are really missed in the base SDK (High Performance Progress Bar anyone?).

AutoCompleteBox
DateTimeConverters
DateTimePickers
ExpanderView
HeadersItemsControl
HubTile
Input
ListPicker
LocalizedResources
LongListSelector
MultiselectList
PhoneTextBox
Tilt
ToggleSwitch
Transitions
WrapPanel

A number of these you simply can’t get by without – the WrapPanel and DateTimePickers are definitely in this group.

http://silverlight.codeplex.com/

Portable Class Library project template

Most Windows Phone 7 apps end up having web services/websites/windows applications that sit side-by-side with them in a solution. Often this leads to code duplication as you keep your common classes in more than one place and struggle to keep them up to date/unit test them.

The Portable Class Library Visual Studio add-on solves this by offering a class library project type that is compatible with any .Net project (ASP.Net, WPF, Silverlight, XNA). I’ve also tested this in multiple build-server environments without problem.

Why this isn’t included as part of Visual Studio out-of-the-box we’ll never know.

I’ve used this in nearly every one of my Windows Phone 7 Projects – I highly recommend it.

http://visualstudiogallery.msdn.microsoft.com/b0e0b5e9-e138-410b-ad10-00cb3caf4981

High Performance Progress Bar

This is now included in the Mango version of the Silverlight Toolkit, but was originally created by Jeff Wilcox from the Windows Phone 7 team.

In simple terms if you use the Progress Bar control in Indeterminate mode (i.e. never ending a-synch waiting) in your Windows Phone 7 app, your Progress Bar is being a complete performance hog – the problem is more finicky in that you will only notice this problem once you’ve actually deployed your app to a real device as it runs fine in the emulator.

The problem is caused by the out-of-the-box Progress Bar control using the Compositor thread exclusively for it’s animation instead of the UI thread.

In the early days of Windows Phone development this was a gripe that a number of developers moaned about until Jeff came out with this replacement control.

http://silverlight.codeplex.com/

http://www.jeff.wilcox.name/2010/08/performanceprogressbar/

Metro Grid Helper

Another must have tool from Microsoft/Jeff Wilcox is the Metro Grid Helper.

The Windows Phone 7 team have definitely created something amazing with the Metro design styling that Windows Phone 7 comes with, but like nearly all design styles Metro looks most polished when everything is perfectly in symmetry.

For Metro this means 12px spacing everywhere, run on button text, and overall a simple rule of living and dying by alignment.

The tiny Nuget package MetroGridHelper (its only a single class!) serves these needs perfectly by simply overlaying a 24px x 24px grid with 12px spacing over your app to make fixing alignment issues and conforming to the Windows Phone 7 Design Guidelines dead simple.

Jeff released the first version of this at the end of 2011 and it hasn’t left my toolbox since.

http://nuget.org/packages/MetroGridHelper

Windows Phone Test Project

The unit testing tools available to Windows Phone 7 developers aren’t really there at all out-of-the-box.

A fellow Aussie Jake Ginnivan, noted this and created a project template that allows you to use either the Silverlight Unit Testing Framework or the WindowsPhoneEssentials.Testing Framework to unit test your Windows Phone 7 code.

I am still yet to see many developers discussing Unit Testing in the Windows Phone 7 world – mostly because a lot of apps written for the Windows Phone 7 platform are either simple or are written by people new to the .Net space.

I think this is a bit of a shame as i think the MVVM patterns works really well with unit testing, so there should definitely be more engagement in this space over the coming year (hopefully).

Check it out in either the Extension Manager inside Visual Studio 2010 or online:

http://visualstudiogallery.msdn.microsoft.com/6819514d-4bd6-4f31-a231-48c6530ed03b?SRC=VSIDE

Emulator Skins

This last one isn’t really essential but if you are spending a lot of time looking the Windows Phone 7 Emulator you’ll most probably be getting a little sick of the basic “looks like no Windows Phone I've ever seen” skin on your emulator.

This can be easily replaced by any number of other phone skins – many from phones you’ve seen your local Windows Phone 7 owners toting, and even other un-released or soon to be released phones such as the new Lumia 800.

The Windows Phone 7 Emulator Skin Switch has 25 skins to try.

http://wp7emuskinswitcher.codeplex.com/

Jeff Wilcox (tell me about it… it’s like this guy has nothing better to do than create awesome stuff for WP7 devs) has also created a single skin based on the device shown throughout the MSDN site:

http://www.jeff.wilcox.name/2011/12/my-new-windows-phone-emulator-theme/

InTheKnow Google Analytics Version 1.8 Released for Windows Phone 7

2/8/2012 3:26:08 AM

Today marks the release of InTheKnow Google Analytics version 1.8 for Windows Phone 7 – there is a lot of little tweaks that have come about during the creation of this next version, and I'm excited that it is out there and getting great feedback. I spent a large chunk of the downtime in my Christmas break working on InTheKnow, so it is great to finally release the next version. If you haven’t tried the app, and are looking for and easy way to check your website’s Google Analytics on your Windows Phone 7 device, check it out – the trial is free.

Below is a list of a number of the added features and changes in the update.

Mango support for Live Tiles

“… Time to cram that start page with some Tiles*…”

You can now pin multiple Live Tiles to your device’s start screen.

I’ve seen how a lot of apps allow you to pin live tiles, but InTheKnow is slightly different in the fact that it’s main demographic (web developers looking for easy to access stats on their sites) often are complete ‘stat whores’ in the sense that they want to check the stats multiple times a day.

I am often no different, even if i haven’t posted a blog post in a couple of months or created a new site in a while – because of this character trait of mine i wanted to be able to pin not only multiple sites but also multiple time periods for multiple sites.

Now you don’t even need to open InTheKnow to know lots of different stats at the same time.

I have also added support for back Live Tiles, with the addition of bounce rate and time on site to the stats you can see from each of your live tiles for the period.

* Live tiles require cellular data usage to update, pinning many live tiles will obviously increase your data usage – Caveat emptor.

New report: Time on site

There are still a few key reports that I want to add to InTheKnow over time, so I'm excited to add Time on site to the list of reports displayed in the app.

As your site grows, one of the key indicators that your visitors are engaged is the amount of time they spend interacting with your website. There is a good summary on why this doesn’t always show an absolute value over on the Webmasters StackExchange.

Watch out for more reports to come.

New polished look and feel

I’ve learnt a lot about designing and building Windows Phone 7 applications since i first created InTheKnow, and therefore i felt that it was about time refreshing the layout and detail put into the finished product.

Jeff Wilcox has made it dead easy to make beautiful looking Windows Phone 7 apps by releasing both a Metro Design Guide for developers and the MetroGridHelper, both of these helped a lot in polishing the app to where it is in this release.

Have your say

Are you using InTheKnow? do you have some feedback you’d like to let me know about?

If so, why not shoot me an email telling me all about it – I'd love to know.

DevOps DNS for Developers – Now There’s No Excuse Not To Know

3/3/2012 1:06:51 AM

Over the years I have had the luck to work alongside many really smart, switched on people in the development community. I’ve learnt from them many intermediate and experienced programming skills. Generally when it comes understanding the very basis of how the internet functions using DNS, most of these very same experienced developers haven’t got a clue. I wrote this post to hopefully help pay back some of the awesome karma they have earned helping me over the years, by teaching them something in return. Lets learn about DNS.

DNS is a huge part of the inner workings of the internet. Web Developers spend a considerable amount of man hours a year ensuring the sites they build are fast and respond well to user interaction by setting up expensive CDN’s, recompressing images, minifying script files and much more – but what a lot of us don’t understand is that DNS server configuration can make a big difference to the speed of your site – hopefully at the end of this post you’ll feel empowered to get the most out of this part of your website’s configuration.

What I will cover in this post:

How the internet works from a Domain/DNS standpoint.
Tools that you should use when investigating DNS problems or testing configuration.
A basic overview of the main types of DNS records.
Setting up a site from scratch.
Investigating some common problems.

Why does DNS matter to you?

Well it’s simple – if you are a developer it matters to you because:

You are a passionate developer, and therefore knowing this stuff is your job.
You own a website, and up until now your webhost has taken care of your DNS for you – but you need to know what's going on in case something bad happens…
Maybe the webhost you have allows you to manage your own DNS using a web interface, but you haven’t a clue what you are doing.
The DNS that your webhost or ISP offers you is probably not the fastest – if your website grows over time, you probably want to setup your own DNS or manage it through a dedicated service such as DNSMadeEasy, ZoneEdit or DynDNS.

First up: How the internet works (DNS)

If you already know how this works feel free to step ahead.

In really simple terms, when you enter a URL and hit enter, apart from magical unicorns rendering the requested page in your browser window, the interwebs works kinda like this:

You want to visit to a domain name, so your PC first checks its internal DNS cache to see if it’s looked it up recently – if so it uses this record
Your PC then asks your DNS server (probably configured by your router or ISP when you first started your PC) for the IP address of the server hosting the domain name you want to visit.
Your ISP’s DNS server looks up the root DNS servers for the world to find out who takes care of the DNS configuration for the domain you want to visit.
Your ISP’s DNS server then asks this authorative DNS server for the domain name you want’s IP information, fetches it, caches it, and then returns it to your PC.
Your browser connects to this IP address and asks for a web page.

There are a number of different scenarios that play a role in special circumstances with the above but I'm not really going to cover everything in this post.

What DNS does do:

Converts hostnames to IP addresses.
Stores mail delivery information for a domain.
Stores miscellaneous information against a domain name (TXT records).

What DNS doesn’t/cannot do

Redirects users to a different server/site.
Configure which port the client is connecting to (not entirely true; SRV records are used for protocol/port mappings for services).

Tools for the Job

One of the coolest things about the tools you’ll need for this blog post, is where I tell you that independent of which operating system you are using, you almost certainly have everything you need to query and test the DNS configuration of your website installed right now without you even knowing

The Swiss Army Knife of DNS inspection is the command line tool NSLOOKUP. This is installed by default in nearly every OS you’ll ever need it on.

NSLOOKUP on Windows

NSLOOKUP on Unix/Linux/Mac OSX

Another cool thing the usage is the same on most platforms as well.

To run NSLOOKUP simply open a terminal/command prompt and type

nslookup

The first thing you’ll notice about the pic above is that the first thing NSLOOKUP tells me upon launch is the current DNS server that it will use for its lookups.

By default NSLOOKUP will use your current machine’s DNS settings for its DNS lookups. This can sometimes give you different results from the rest of the world as your internal DNS at your place of work/ISP may be returning different results so they can route, say your office mail, to the internal mail server IP rather than the external internet/DMZ IP address.

Lets change this to use Google’s global DNS server to get a better global view (what others see when they surf the web outside my network) on our DNS queries, by typing;

server 8.8.8.8

Now if I query this blog’s domain name “diaryofaninja.com.” (ensure you place the additional period on the end of this query to avoid any internal DNS suffixes to be added) I should get back the A record for my domain; (A records are the default query type used by NSLOOKUP – I discuss DNS record types further in this post below).

An overview of common DNS record types

Below is a simple overview of all the common types of DNS records and some example scenarios.

All records usually share the following common properties:

Value – this is usually the contents of the records. If it is an A record this is the IP address for that A record

TTL – this is the “Time To Live” in seconds for a DNS record and basically means that DNS Clients of Servers accessing the requested record should not cache the record any longer than this value. If this value is set to 3600 this means to cache the returned record’s value for an hour (these values are usually the reason that IT people talk about DNS changes taking “24-48 hours” as these values are usually set quite high on hostnames that are quite static so that they offer the best performance by being kept in cache.

SOA (Start of Authority) records

SOA records (start of authority records) are the root of your domain’s registration. SOA records are created by your domain name registrar in the parent domain’s DNS servers (in the case of a .com domain the SOA record is created in the DNS servers for the .com root domain. In an SOA record the hostnames or IP addresses of your domain’s DNS servers are stored. These tell the internet’s root DNS servers (mother ship DNS servers) where to ask for the rest of your domain’s DNS configuration (such as A, MX and TXT records). When a client (a web browser, a mail server, an FTP client etc.) wants to connect to part of your website, it asks the locally configured DNS server for the record – the server in turn looks for the SOA records for your domain so it knows which DNS server to ask about it.

Consider these records as the source of “which DNS server stores all the information about the website I want to look up”.

Hostname (A and CNAME) records

A records store information about a hostname record for your domain name. These list the IP address that a client should talk to when using a certain hostname.

If you had an address of http://mywebsite.examplecompany.com into your web browser this would refer to the A record “mywebsite” on the domain “examplecompany.com”.

If you have multiple A records with the same hostname, clients will receive a list of all the records. The order of this list will change with iterate each time you query the DNS server – this is called round-robin DNS and it a simple way to spread load across multiple servers .

AAAA records are the same as A records, only they stores the 128bit IPv6 address of a server instead of the IPv4 IP address – as the world shifts to using IPv6 these records will gain more relevance, but if your webhost supports IPv6 its worth setting these records up now, so that any visitors using IPv6 can access your website.

A CNAME record (Canonical name) is basically an alias for an A record. This tells whoever is asking, that the DNS information for the requested hostname is stored in another record somewhere else on the internet. This other record might not even be on the same domain name or on the same DNS server. CNAME’s are very powerful as they allow you simplify your domains DNS records by centralising the information somewhere else. ISPs and webhosts commonly use CNAMEs to centralise the DNS configuration storage for things like mail or web server’s by allowing you to keep all the configuration details on a parent domain name.

It is important to note that root records for a domain name (I.e the empty A record for mydomain.com) cannot be a CNAME. The simple hard and fast reason why, is that CNAME’s cannot live on the same node in a DNS forest as any other type of record – because the very nature of a CNAME record defines that all configuration for that node is stored somewhere else, and given you store other information at the root of your domain other than your A record (MX records for mail etc.) this would break every other record’s functionality. This is mentioned specifically in the RFC for DNS, section 3.6.2.

An example of CNAME usage, is when most webhosting company web servers have a hostname such as web0234.mywebhost.com

When setting up your website, your webhost might for instance make the “www.yourwebsite.com” record for your website a CNAME that has the value “web0234.mywebhost.com” so that when trying to access “www.yourwebsite.com” DNS clients look up the IP address for “web0234.mywebhost.com”. This makes their life easier if the IP address for this web server changes, as they only have to update a single DNS record, instead of updating all their clients DNS records.

To reiterate this to make it crystal clear:
CNAME’s are not a redirection. They are a reference pointer for a hostname. All they tell DNS clients, is that the configuration information for the hostname being queried is the same as can be found by querying the other hostname.

Illustration – Visiting a website

In the case that you want to visit www.google.com your computer does the following:

Using the local machine’s DNS client your operating system talks to the locally configured DNS server for your local network/ISP’s network.
This DNS server inturn looks up the DNS server for google.com by first looking up the SOA record for google.com and then connecting to the DNS server listed.
Your local DNS server then asks the DNS server for google.com for the A record listed for www – the google.com DNS server will return an IP address for www.google.com. Your ISP or local network’s DNS server, along with returning it to you, will then cache this record for as long as the TTL (time to live) property of the record says.
Your browser then connects to this returned IP address listed on port 80 and asks for the web page.

All of the above happens in milliseconds – but you can understand that if the google.com DNS server is slow in responding this negatively affects your browsing experience.

A records and CNAME records have a TTL (Time To Live) property to indicate how long they can be cached for.

Mail (MX)

MX records are the internets way of telling mail where to be delivered. They list the hostname or IP address of the mail server that handles mail for a given domain name. If a mail server is looking to deliver mail to “examplecompany.com” it will look up the MX record for this domain.

MX records have both a TTL (Time To Live) and a Priority (a weighting to give the order in which they should be looked up).

Illustration – Sending an e-mail to a friend

In the case that you send an email to your friend at myfriend@otherexamplecompany.com your local SMTP mail server (usually at your ISP) does the following:

Your mail server connects to its local network/ISP’s DNS server and asks for the MX record for otherexamplecompany.com.
Your local DNS server or ISP’s DNS server looks up the SOA record for otherexamplecompany.com and then connects to the DNS server listed.
It asks for the MX records for this domain and is returned a list of hostnames.
it grabs the first hostname from the list (order in ascending order by Priority), runs a second query for the IP address of this mail server and returns this IP address to your mail server.
your mail server then connects to this IP address on the SMTP TCP port 25 and delivers your mail.

Text Records (TXT)

TXT records are a powerful addition to the DNS standard that allow the storage of miscellaneous information for a hostname. Many web developers, system admins and the like use TXT records for the storage of information such as SPF records and DKIM public keys.

TXT records have a TTL (Time To Live) property to indicate how long they can be cached for.

Name Server Records (NS)

Name Server Records are placed in your domain’s DNS when you wish to store the configuration of part of your domain’s DNS on a separate DNS server. This can be very handy if you want to give control of a subdomain to another person/entity.

i.e. my site is www.widgetsareus.com and I manage all of the DNS for this domain, but I would like support.widgetsareus.com and any child sub domains of this domain to be managed by the company we outsource all of our customer support to – therefore I have setup an NS record for support.widgetsareus.com to point at our support partner’s DNS servers.

Setting up a domain from scratch

If you are setting up a domain you’ve just purchased from scratch you’ll need to do the following:

Setup your website (A records)

Setup a DNS server to store the configuration for yourdomain.com
This might be at your webhost, or might be a third party service such as DNSMadeEasy, ZoneEdit or DynDNS.
Set the Nameserver SOA records for your domain name to the above DNS server’s IP address or hostname (at your domain registrar)
Create a new root record to point at your webserver’s IP address (this is simply an A record with an empty hostname) in your domain name’s DNS forest.
Create a new www A record that points at your webserver’s IP address in your domain’s DNS server
Setup your webserver’s website to listen for the host-header of your domain name (IIS calls this a “binding”).
Test your DNS as below.
Try and access your site in a web browser.

Testing your website’s A record

In a command prompt/terminal type NSLOOKUP

Enter “yourdomainname.com.” (including the extra period on the end) and hit enter

Check that the returned record value/IP address is that of your web server.

Remember to do the same for “www.youdomainname.com.” if you also use www. in your domain name.

Setup your website’s mail (MX record)

Setup a DNS server to store the configuration for yourdomain.com (Follow steps 1 and 2 above from your website if you haven’t already).
Create a new MX record that points at your mail servers IP address or hostname.
Setup your mail server to listen to receive mail for yourdomain.com
Test that all the above is setup correctly using nslookup as per below.
Try and send and receive email to and from your domain name.
Setup SPF records, to verify your mail server’s ability to send mail on behalf of your domain name

Testing your website’s MX record

In a command prompt/terminal type NSLOOKUP

Enter “set type=mx” and hit enter. This set the query type to MX records.

Enter “yourdomainname.com.” (including the extra period on the end) and hit enter

Check that the returned record value/IP address(es) is that of your mail server.

Investigating Common Problems

How do I check what DNS server is authorative for my domain name?

You’ve set up your websites DNS, everything is fine; then one day, everyone visiting your site is directed to a site that isn’t yours!

To check which DNS server is authorative for your domain name, first open a command prompt or terminal.

Type “NSLOOKUP” and hit enter

Type ”set type=ns” and hit enter. This sets the query type to NS (NameServer) records.

Type “yourdomainname.com.” and hit enter (make sure you put the extra dot on the end.)

Confirm that the nameserver’s returned are yours.

How do I check what IP address my site is currently pointing at?

In a command prompt/terminal launch NSLOOKUP

Enter “yourdomainname.com.” (including the extra period on the end) and hit enter

Check that the returned record value/IP address is that of your web server.

Remember to do the same for “www.youdomainname.com.” if you also use www. in your domain name.

What is split DNS?

Split DNS is when you run a separate DNS forest for a domain name both on your external DNS servers (for everyone else to see) and also internally for staff or local users to see.

This allows you to do things like:

Ensure local users talk to your mail server (or any other internal server) using the internal IP address, and internet users talk to your mail server’s external DMZ IP address.
Block access to certain sites by giving incorrect or different DNS results for these site’s domain names. This if often how many net nanny etc softwares work.

For some users my sites seems to be served from a different address – how do I check “what the world sees” vs. “what I see”?

Many things can occur that result in some people seeing different DNS results to others:

Your ISP/company’s DNS server may have an older cached record to the current live record
Your local computer may be caching the DNS record you are requesting
Your local DNS server may be fetching the records for your domain from a different authorative DNS server than the rest of the world.

How do you investigate these things?

The easiest way to investigate these things is to query an external DNS server that you know is good for the records you want, to get a better idea of how the rest of the world sees things.

A really good server that is easy to remember are the ones owned by Google. The primary and secondary DNS server for Google’s Public DNS system are “8.8.8.8” and “8.8.4.4” respectively.

You can use whatever DNS servers you think are more likely to see the correct values.

To do this, open a command prompt/console.

Type “NSLOOKUP” and hit enter

Type “server 8.8.8.8” and hit enter. This sets the DNS server we will query to the Google Public DNS server’s address.

Type “yourdomainname.com.” and check the resulting record values.

What ASP.Net MVC Developers Can Learn From GitHub’s Security Woes

3/11/2012 7:01:00 AM

Over the last week a few stories have moved through the Ruby On Rails and wider development community as one of their shining stars, GitHub was hacked to draw attention to some of the weaknesses that can come about from ROR’s convention-based model binding. The interesting thing about the security hole found at GitHub is that it is not necessarily limited to Ruby On Rails, but often comes from using a framework that supports model binding out of the box without understanding the security limitations up front. It also brings a question to the fore: Is it the role of framework developers to force any security configuration to be the default instead of being explicitly applied?

A ROR developer Egor Homakov, found a configuration vulnerability in the ROR framework’s scaffolding. This vulnerability affected all ROR sites that had been developed using the basic scaffolding that comes out of the box without consideration for security issues, and its implementation of relationship links between data access classes.

Initially Egor, tried to bring attention to this by posting an Issue to the Ruby On Rails Github site. After not receiving the response he thought the Rails community should have been giving to such a wide affecting security configuration bug, posted a message with a timestamp 1,000 years in the future and then added his private key to the Rails group and uploaded a (completely innocuous) file to the Rails Git repository – all to prove his point.

What I'll show you below, is the problem that GitHub had, how this can also occur in a hypothetical situation in ASP.Net MVC, and what you can do about it.

Let’s Run Over The Problem

Ruby On Rails is a pretty slick Ruby framework for building websites fast, and one of the main ways it does this is by taking a database and scaffolding a site from it, with everything required to get up and running.

In Ruby On rails, If you have a data model that looks like this (the foreign key would actually not be included in your model object, ROR injects this for you):

When you scaffold the tables you get code that looks like this (I've removed any code we aren’t going to talk about):

# == Schema Information
# Table name: users
#
#  id                         :integer(11)     not null, primary key
#  email                      :string(255)     
#  username                   :string(255)     
#  password                   :string(255)
class User < ActiveRecord::Base
  belongs_to :securityroles

  validates_presence_of :email, :password
  validates_uniqueness_of :email

  ...
  #class scaffolding code
  ...

end

# == Schema Information
# Table name: securityroles
#
#  id                         :integer(11)     not null, primary key
#  roleName                   :string(255)     not null
class SecurityRole < ActiveRecord::Base
  has_many :users

  validates_presence_of :roleName
  ...
  #class scaffolding code
  ...

  def update
    @user = User.find(params[:id])
 
    respond_to do |format|
      if @user.update_attributes(params[:user])
        flash[:notice] = 'User was successfully updated.'
        format.html { redirect_to(@user) }
      else
        format.html { render :action => "edit" }
      end
    end
  end
end

You’ll notice above that the main thing to be concerned about with this scaffolded code is the part where it says this:

def update
  @user = User.find(params[:id])
 
  respond_to do |format|
    if @user.update_attributes(params[:user])
      flash[:notice] = 'User was successfully updated.'
      format.html { redirect_to(@user) }
    else
      format.html { render :action => "edit" }
    end
  end
end

What ROR is doing here is a feature they call “Mass assignment”.

What the above code is doing:

Pulling a User record from the database using the incoming parameter id to find it.
Binding all the other incoming Request parameters to the data model loaded from the database and updating them.
Saving the new model back to the database.

Egor’s attack on GitHub, was dead simple in that he simply added a new field to the page that submitted new SSH keys for a user that had the field name of id, and he then proceeded to make this field’s value the id for the Ruby On Rails project owner. This caused the above default scaffolding code to save his GitHub SSH Private key not to his own GitHub account, but to the Ruby On Rails GitHub Account. Sneaky. Also a very basic oversight by GitHub.

To be fair to Ruby On Rails, there is a way to stop this kind of attack by marking your class with the property attr_accessible nil to turn off mass assignment support for a class unless specified so you can white list only the fields you want to bind by hand – the problem with this feature is that it is not the default. You have have to do this manually – and we all know how easy it to skip or forget about these things in the rush to meet a deadline.

In ASP.net MVC this ROR functionality would be similar to our model binding. Obviously the code that the ROR scaffolding tool creates is a lot more functional than we get out of the box with ASP.Net MVC, but the CRUD functionality they are creating is very similar across all web platforms. In ASP.Net MVC you have to write your own CRUD code to add, edit and update records and ASP.Net MVC tries to make this simpler by adding Model Binding into the mix.

In some ways the security concerns this raises are greater because as if you are a junior developer you might not be clear on any potential nasties that can come about if when you write insecure code, or what this even includes.

What This Behaviour Look Like In Practice

To take a look at what this would entail in ASP.Net MVC let’s think about how the above Ruby on Rails data model application would actually play out in real life using ASP.Net MVC.

You would most probably have a membership based site where users log in, have different levels of access to things, and can change their settings.

If you weren’t switched on and you were new to ASP.Net MVC and web development it would be easy to write a user settings page that looked like this:

@using (Html.BeginForm()) {
    @Html.ValidationSummary()
    <div>
        <fieldset>
            <legend>Account Information</legend>

            <div class="editor-label">
                @Html.LabelFor(m => m.Name)
            </div>
            <div class="editor-field">
                @Html.TextBoxFor(m => m.Name)
                @Html.ValidationMessageFor(m => m.Name)
            </div>
            
             <div class="editor-label">
                @Html.LabelFor(m => m.Email)
            </div>
            <div class="editor-field">
                @Html.TextBoxFor(m => m.Email)
                @Html.ValidationMessageFor(m => m.Email)
            </div>
            <p>
                <input type="submit" value="Change Account Details" />
            </p>
        </fieldset>
    </div>
}

And your settings controller could look like:

[Authorize]
public ActionResult ChangeAccountDetails()
{
    var dbUser = DAL.UserRepository.Load(x => x.Username = User.Identity.Name);
    var model = new UserModel(dbUser);
    return View(model);
}
[Authorize]
[HttpPost]
public ActionResult ChangeAccountDetails(UserModel model)
{
    DAL.User dbUser = DAL.UserRepository.Load(x => x.Username = User.Identity.Name);
    if (TryUpdateModel(dbUser))
    {
        DAL.User.Save(dbUser);
        return RedirectToAction("ChangeDetailsSuccess");
    }
       
    ModelState.AddModelError("", "Something went wrong");
    return View(model);
}

What’s Wrong With This Code?

If you’re stumped looking at the above code to find any issues, it’s time to start thinking about one of the golden rules of web development security; White Listing.

The exact same problem that affects the Ruby On Rails community in the GitHub hack with their mass assignment “bug” (if we even call it that), is that the above ASP.Net MVC code blindly binds any incoming request parameters to our database model.

This is exactly the same problem that allowed Egor to be able to hack GitHub:

We don’t specify what incoming parameters we actually care about.

This can lead to all manner of implicit security or data quality issues on your sites, because users can potentially hijack your own code for other purposes – it’s kind of like the modern equivalent of SQL Injection, but through binding code!

How could we hack the above piece of MVC code?

The problem that could affect any ASP.Net MVC site that isn’t verifying all incoming user submitted data, usually comes about not when the site is used as intended – but when new parts are added to the request.

The site gives us data.
We send the updated data back.
But we also added something sneaky along for the ride.

Lets change our Account Settings page, and add a new field.

@using (Html.BeginForm()) {
    @Html.ValidationSummary()
    <div>
        <fieldset>
            <legend>Account Information</legend>

            <div class="editor-label">
                @Html.LabelFor(m => m.Name)
            </div>
            <div class="editor-field">
                @Html.TextBoxFor(m => m.Name)
                @Html.ValidationMessageFor(m => m.Name)
            </div>
            
             <div class="editor-label">
                @Html.LabelFor(m => m.Email)
            </div>
            <div class="editor-field">
                @Html.TextBoxFor(m => m.Email)
                @Html.ValidationMessageFor(m => m.Email)
            </div>
            <!--                 
                #HACK ATTACK#                
                THIS FIELD WILL SET OUR ROLE ID TO "1" 
            -->
            <input type="text" name="SecurityRoles_Id" value="1"/>
            <!-- -->
            <p>
                <input type="submit" value="Change Account Details" />
            </p>
        </fieldset>
    </div>
}

The site gives us our page.
We add a new field, named SecurityRoles_Id to the page using Firebug etc.
When we submit the page back the site would set our security role foreign key to 1.

You can see from my above view code that if I added this field in Firebug or developer tools and submitted it back to the site, I would have changed my role in the site’s database to whatever role was stored in the database’s roles tables with the Id of 1.

Think back to when you first set up your database – What was the first user (record number 1) you added to your database, an Admin? What was the first role (record number 1) you added to your Security Roles table, Admin?

This problem is ever simpler if you have a user table that simply has a Boolean field called something like “IsAdmin” to define the users permissions as an attacker wouldn’t have to guess what the id of a nice juicy role is set to.

I understand that looking at the above attack you have to know that my database field was called securityroles_id but if you ask anyone working in web security they’ll tell you have unless a site has also locked down all its error messaging or tracing this can be sometimes easy enough to find out using other attacks first to probe your data model and review the errors thrown.

Batten Down The Hatches

Whitelist your submitted data

When visitors to your site submit data back to you, always white list the fields you are saving to the database. This means manually bind or copy only the fields you want to save from the request – don’t just bind the whole model to a database record.

Use model binding carefully with sensitive data to avoid any accidental oversights while developing your application.

ASP.Net MVC supports protection against this in a similar way to Ruby On Rails’ attr_accessible nil by enabling your controller action to use the Bind Attribute [Bind(Include=*explicit properties here*)] when defining your incoming model binding. Just like ROR however, this is not turned on by default…

public ActionResult ChangeAccountDetails([Bind(Include = "Name,Email")] User model)
{
    ...
}

If your model class has many fields you can work backwards, and exclude properties from binding instead;

public ActionResult ChangeAccountDetails([Bind(Exclude = "SecurityRoles_Id")] User model)
{
    ...
}

You can also use the additional overloads of the TryUpdateModel method to define what objects to bind, or not bind from the FormCollection if you are using this method of model binding/validation.

Include;

[Authorize]
[HttpPost]
public ActionResult ChangeAccountDetails(UserModel model)
{
    DAL.User dbUser = DAL.UserRepository.Load(x => x.Username = User.Identity.Name);
    string[] includeInModel = new[] { "EmailAddress" }; 
    if (TryUpdateModel(dbUser, includeInModel))
    {
        DAL.User.Save(dbUser);
        return RedirectToAction("ChangeDetailsSuccess");
    }
    ModelState.AddModelError("", "Something went wrong");
    return View(model);
}

Exclude;

[Authorize]
[HttpPost]
public ActionResult ChangeAccountDetails(UserModel model)
{
    DAL.User dbUser = DAL.UserRepository.Load(x => x.Username = User.Identity.Name);
    string[] excludeInModel = new[] { "SecurityRoles_Id" }; 
    if (TryUpdateModel(dbUser, string.Empty, null, excludeInModel))
    {
        DAL.User.Save(dbUser);
        return RedirectToAction("ChangeDetailsSuccess");
    }
    ModelState.AddModelError("", "Something went wrong");
    return View(model);
}

Also, if you have complex types in your view model (Entity Framework objects etc.) you should abstract your data layer away into a separate project. Other than the fact that you should never have complex types of dependencies in your models, doing this ensures that even if you do step on a land mine by accident, you aren’t talking directly to the database and can have a little more control. If you are only new to this additional separation and are cringing at the thought of all the additional code you’ll have to write, take a look at something like AutoMapper to do this for you (ASP.Net MVC 3 has the TryUpdateModel method that I used in my first example that does this for you as well – albeit with less control).

public ActionResult ChangeAccountDetails(UserModel model)
{
    //create a new User object
    DAL.User u = new DAL.User();
    // map the incoming model to the user object
    AutoMapper.Mapper.Map(u, model);

    //
    // Some code that attaches our object 
    // to the current record in the DB that 
    // matches User.Identity.Name
    //
    DAL.User.Save(u);
    return RedirectToAction("ChangeDetailsSuccess");
}

Turn Off Any Exception Signalling

Ensure that all tracing and exceptions are turned off on your site to avoid probing. This means turning friendly error messages on to avoid any exceptions being handed out to attackers wanting to test the waters, ensuring tracing is turned off, and take a read of Troy Hunt series on OWASP security vulnerabilities that can often occur during application development without you being aware.

Machine.config

<configuration>
  <system.web>
    <!-- turn on retail mode -->
    <deployment retail="true" />
  </system.web>
</configuration>

Web.config

<configuration>
  <system.web>    
    <!-- turn debug mode off -->
    <compilation debug="false"/>    
    <!— turn On custom errors -->
    <customErrors mode="RemoteOnly"/>
    <!-- turn tracing off (this is set to false by default) -->
    <trace enabled="false"/>
  </system.web>  
</configuration>

Summary

While more senior web developers working on the ASP.Net MVC stack may think the issues I've raised are too hypothetical and easy to overcome, I can assure you from much experience working with developers of all sorts of skill levels over the years that these types of inherent “you’d never do that, it’s silly” problems are usually the ones I’ve see most.

GitHub’s problems were more related to the way ROR does it’s scaffolding, but it just goes to show that development techniques that save time by writing the code for you (such as ASP.Net MVC’s Model Binding) can often lead to a disconnect between the developer and his application’s code – if a large company like GitHub can miss these problems, its proof that these problems are more of a commonplace occurrence than we’d all like to think.

When saving any user generated data, be sure to only save what you intend to for a given action, and ensure they have permission to do so before putting saving it to your database.

The Line Between Insanity and Genius. Where Do You Draw Yours?

3/25/2012 5:35:00 AM

As Software Developers, our passion can sometimes be all encompassing. You can find a new language or framework and something inside you lights a spark and the obsession begins – it’s part of what makes a great scientist, developer, engineer or doctor. It can also be a part of what drives people insane. Recently I've wondered where the line is, when its time to back off, or even if you should.

Lately I've had a few moments of introspection which some may view as disturbing, and others may celebrate: the realisation that a large part of my career’s success, and my personality could easily be misconstrued as mental illness if looked at under the wrong light.

Obsession – It can be never ending

I can find parts of technology or programming that can consume me. It'll be all I think about night and day for weeks.

Remember that time you asked me a question and I might not have responded straight away? I was probably thinking about that new framework, design pattern, process or technology I was hooked on. Thinking about my social life it has probably limited my interaction with the world multiple times over the course of my life.

When it comes to work, it can be a dangerous thing for totally other reasons. When do you stop pushing for perfection to save wasting money, resources or staff members. How do you explain this to non-technical members at your workplace.

Not too long ago in Scott Hanselman and Rob Conery’s “This Developers Life” episode on “Obsession”, they talked to Rob Sullivan and Rory Blyth about what drives them as developers, and how they became obsessed with technology – and how they sometimes find it hard to unplug; hard to even get themselves to step away from a problem.

For me I find this very inner torture happening all the time.

The most poignant quote I took away from that podcast (I listened to it again recently), was Rob Sullivan’s quote:

“…

As I’ve looked back and thought about everything I've done over the last couple of years, I had a conversation with myself… So I asked myself a question in my head, ‘What do I want to do for fun’, and then i got into this very Socratic Dialectic interaction with myself about ‘Ok, fine so what is fun? Define fun.’ and I could not figure out what fun was.

And I think that’s what happens in the Tech industry. When you are actually into what you are doing, you are having fun, but you are also working; and you never, ever stop. You lose perspective.

…”

This is exactly how I feel sometimes.

I feel that sometimes I want to stop my brain from churning over and over on a coding problem I’m trying to solve, a topic of conversation I’ve had with someone, or an opinion I just read on a blog. Sometimes I want to switch that part of my brain off so I can relax.

And it doesn’t end with Tech for me – I have this “obsession” often with everything that I do. I have to figure out or learn about everything I interact with – and Google has made it really easy for people like me to get their next fix.

“How does the body metabolise different foods?”

“What frequency do bats use to see in the dark?”

“Do red hair people really need more painkillers than others?”

It never ends. I often cannot stop this type of thinking and have had to create walls and rules to stop it and live a normal life – I don’t take a smart phone out to social occasions anymore.

This type of obsessive questioning and reasoning inside my head gets me to thinking – “Is there something wrong with me?, Am I starting to go all ‘Beautiful Mind’ on myself?”

From the small conversations I’ve had with other people in the Tech world, I am not alone here. A large section of us have stopped and questioned ourselves or our drivers.

Where is the line?

The hard part I find is that I come from a family that has dealt with mental illness. That thing that no one ever talks about, I dealt with through my mother, who was Schizophrenic and therefore I would think to myself that I think about it more than most. My mother was passionate and intelligent beyond measure – and music was her passion. It was everywhere – it drove mood, it was mathematical, and it was always to be in search of.

How am I and most really passionate people in the Tech industry any different in some ways?

When you stay up late into the night trying to finish a game, or complete the next feature in your project; or totally rewrite that part of a project – Do you actually have control to stop? Do you deep down actually want to stop? And at what point is this lack of control a bad thing?

What about when you finally have time off work, and all you want to do is start a new Open Source project, or play with a new language (you’re meant to be on holidays remember?)

Others in our industry

Since Steve Jobs died recently, there where many stories and rumours of how he dealt with work life at Apple and his obsession for perfection. If Steve had another obsession that wasn’t creating number one selling consumer devices, would people have the same feelings for his life’s deliverables?

There has often been questions of Mark Zuckerberg’s potential case of Asperger’s syndrome. This very same article talks about tech and it’s connection to these types of obsessive and rigid character traits - however science likes to define them. There has likewise been suggestion that many other people in tech, Bill Gates included, might show symptoms of Asperger’s.

Regardless of definition, I know I'm not alone here in thinking that sometimes my obsessive or focused behaviour might not be normal.

Where do you define what is healthy when having a passion for something – Is this something society defines to you, or do you set the goal posts? And if you, how do you stop yourself from moving the goal posts through self-justification?

If you are passionate to the point of changing parts of your life to suit your passion – where is the line between passion and craziness, insanity and genius? And given my experience; If you ever think you cross the line – how do you have the strength to take heed?

Picture of the Sistine Chapel by féileacán licensed under Creative Commons.

I’m a Junior Developer – You probably are too

3/27/2012 6:37:58 AM

Part of my job is hiring people to build websites for advertising clients. Most of the things that we build don’t compete with brain surgery for complexity, but as anyone knows when working in software, having skilled people working on simple problems often leads to scalable, well built solutions – I hire accordingly. The problem is how do you define "skilled" and does it even have meaning in software out of the context of a company’s needs?

I have been quite lucky that early in my career I realised that no matter how skilled I got at my craft, there would always be someone on a whole other level. Crossing developers with egos aside, attending conferences, user groups and development events is one of the easiest ways to reset your expectations surrounding skill level, and find this out early.

I am a Junior Programmer

I have been developing software since I was 10. Commercially since I was 14 and 9 months (this is the legal age in Australia to work). I have had a go at creating many different types of software in many different programming languages, more recently on the Microsoft stack.

I’ve lead teams, travelled for work, had high risk, low risk, sexy, boring, you name it programming jobs over the years.

All this has made one thing very clear to me:

I know nothing.

Our industry is so vast, and different technologies and methodologies are so numerous that this single piece of information should go with you everywhere you travel.

How about you?

When applying this to hiring though, things are never as clear. Nowhere is this more evident than working in a Sydney advertising agency, and being tasked with hiring web developers.

You see in the land of web development in Sydney, the age of the Brogrammer/Coder is upon us.

Recently I have been tasked with hiring a new senior technical member of staff. I literally did 30 or more phone interviews, many-many in person interviews and the experience left me feeling quite dis-enchanted with the Sydney .Net web world. The amount of 10+ year experience huge salary swinging web developers I saw who couldn't:

Describe how POST data was submitted to a server by a browser.
Explain a number of HTTP status codes (except maybe 404 and 500).
Explain SOLID or name a design pattern.
Explain ways to improve a pages load speed or user experience.

You may think I am being being aggressive in suggesting this, but if you can’t answer the questions above there are a lot of people who wouldn’t think of you as a Senior Web Developer (maybe we can rule SOLID out… but you understand my point). However what do you call yourself if you have been building websites for 10 years in ASP.Net (or PHP, or Perl, or anything…) and you need a title?

This got me thinking though: how do you define seniority or skill in Software Engineering terms. Especially when hiring.

So many people can have simple roles for long periods of time, where they aren’t necessarily exposed to new principles or changes in their industry – do these people deserve high 6 figure salaries and “Senior” or “Architect” in their job titles?

This is where I think it all comes down to a single word:

Context

If you have been building applications, websites, or *whatever* for many many years, you would hope you would have become considerably good at it. This doesn’t make you a senior developer, and this I think often gets lost on most of our ilk. It does make you good at what you have been doing.

But my view on people compared to your view on a candidate and their skills might be totally different – because I am totally blinded by the context for which I need from my new team member. My definition of senior will be totally different from that of someone hiring at a place that builds landing systems for spacecraft; or someone working on small business accounting packages.

In my hiring case, I was looking for someone who had a fair bit of experience building websites in ASP.Net and c# among other things. Not someone to build the next Google. However our industry only thinks of people in terms. “Junior ASP.Net developer”, “Senior Ruby programmer”,”Mid level PHP developer”. I don’t think this describes people’s experience or knowledge levels well enough.

Usually this equates to someone who is new, has been around for a while, or is an “old salt” at whatever they were doing – maybe building “Hello World” websites over and over… Who knows?

Maybe it’s time to rewrite how we describe people in our industry. Something similar to Scott Hanselman’s use of “FizzBin” in technical support calls might just do…

Things I Missed So You Won’t Have To – WP7 SDK

4/2/2012 7:04:13 AM

During my development journey with Windows Phone 7 , there have been a number of things that I overlooked when writing your first app – things I wished I’d kept front of mind. None of these items are necessarily difficult or complicated things to cover off, so I thought I'd mention them here to save you the trouble, and at the same time show you ways to overcome each of them.

Globalisation

Windows Phone 7 devices have been released to a total 63 different markets at the time of this post’s writing, and while we might not sometimes talk about it too much the majority of the world doesn’t speak English.

Even more relevant when writing apps that process third party data, lots of countries also don’t use the same date or decimal system as the US or British do – and when parsing dates or numbers in a local region this can easily break your code.

If you are in France or a large percentage of Europe for example, instead of One Hundred Thousand being written as “100,000.00” as in the US it is instead written as “100 000,00”.

The way to get around this depends on whether you are receiving data in a foreign region, or sending data in a foreign region.

Below is an example piece of code to represent this problem (This is not how you should your app because it is best practice to keep the culture local and instead change how you parse).

// Switch the current culture to French to simulate a French User
Thread.CurrentThread.CurrentCulture = new CultureInfo("fr-FR");
Double Number = 123.00;

// This will be "123,00"
string Output = Number.ToString();

// This will be "123.00"
string InvariantOutput = Number.ToString(CultureInfo.InvariantCulture);

This means that if you are parsing Strings that are from the US to Ints or Doubles (such as those fetched from remote American APIs such as Twitter or Google) and your users are in France, your app will throw an exception as the number format won’t be properly recognised – you need to either specify the culture of the incoming string or specify the culture as Invariant Culture (defaults to United States).

Take the following example run on a US based device. Here we take a set the culture to French (to simulate a French user) and then successfully parse the US/British number string.

Double Number = 123.00;

// This will be "123.00"
string Output = Number.ToString();

// Change the current culture to simulate a French User
Thread.CurrentThread.CurrentCulture = new CultureInfo("fr-FR");

// This will throw an exception
Double FrenchParsedNumber = Double.Parse(Output);

// This will not throw an exception
Double FrenchParsedNumberInvariant = 
    Double.Parse(Output, CultureInfo.InvariantCulture);

All of the above examples also definitely apply to Parsing of Dates. There is a good write up on MSDN showing how to do this with DateTime format providers.

Also, You can test your application under different conditions by changing your emulator’s regional culture.

Localisation

While this is not really a problem with your app per se, while we are on the topic of foreign users it’s worth mentioning that because very soon only a tiny percentage of the market will be English speaking countries, separating your text content into international resource files will make selling in these markets a lot more lucrative – and anyone who’s done this before will tell you it’s a lot easier if you do this from the start, even if the only language resource you have is English to start with.

Steps to add local language support to your Windows Phone 7 app pages (Based loosely on this MSDN article)

Create a new Resource File in your app called Resources.resx

Add a new file for each additional language you want to support (i.e. named Resources-FR.resx for French, Resources-ES.resx for Spanish).

The MSDN article linked above shows you how to create the LocalizedStrings class, however they don’t go on to show you how to use it for DataBinding, so lets solve that.

WPF/ASP.net and Silverlight handle data binding differently than Winforms. Silverlight uses the term Resource to refer to files that use the the Build Action of "Content”, as these files get wrapped up into the .XAP file similar to files with the Build Action of "Resource” get embedded into the .Dll assembly.

I found that instead of using the Text="{Binding Path=resourceFile.resourceName, Source={StaticResource Localizedresources }}" XAML syntax it was easier to follow the steps below:

Open your primary XAML page (usually MainPage.xaml) in the Visual Studio
Open the properties for the PhoneApplicationPage and set the DataContext to be Application.Resources –> LocalizedStrings. NOTE: if you already are using a DataContext object, then you should integrate the LocalizedStrings class into that object so that it has localization support.
Once the Page’s DataContext has been set you can change the data binding for any control on the page by simply selecting the property (ex: text, checked, etc) and selecting “Apply Data Binding…”, and setting the Path to LocalizedResources.BtnText or whatever the resource key that you are trying to bind to is.

Remember: the Application Bar is not a Silverlight control and therefore can’t have resources bound to it in the same way. To achieve this you will have to create or update the Application Bar at runtime.

Wi-Fi Internet browser login problems

This is another problem that is far from 100% clear at first.

Your users will probably come across bad data while connecting to wireless access points that require a browser login (such as a hotel or library). Usually the device should kick up a session of Internet Explorer to take care of this, but there are a number of situations where this won’t help you:

When your background task runs to update a tile and the user hasn’t logged in yet.
When the user ignores the browser login page or is interrupted while using it (phone call etc.) – this leaves them in a state of “online” but not net connected.

Quite often these browser-login walls for Wi-Fi will simply return the login page for every request. This means that you application will think its speaking to the internet (or Twitter or whatever API you are using) but instead will get back bad data or a 3XX redirect response code.

What does this mean for you?

It means that when pulling down data from an external internet source, make sure to the following:

White defensive code for WebExceptions such as Try Catch statements (as long as you only catch the exceptions you are looking for – not just any exception).
If accessing a data source with a known or documented schema (such as a third party XML API), validate returned data against a schema file (in XML’s case, use an XSD to validate the data format before accessing it or deserializing it). This will ensure that even if data is returned, you know its what you were actually looking for.

Always assume the data that your app will be getting back from the internet is invalid.

Network Interface Online Without Internet

Another problem that can affect your users and leave them thinking bad things of your app is when their devices think they are connected to the Network but don’t necessarily have internet access. This can happen in areas of bad coverage, or when their carrier has internet connectivity problems.

Ensuring that the device has Network connectivity won’t fix this problem as your app will still timeout if it can’t access the internet.

There are two ways to minimise this problem:

While your app is running, every few minutes download a small text file, similar to Microsoft’s NCSI service and use this to establish Network Connectivity (you could even use the MS URL to save yourself time).
Set timeouts on your app’s network downloads to a lower figure than the default of 100 seconds – the standard Windows Phone 7 SDK HttpWebRequest doesn’t support the Timeout property, so the guys over at WP7Contrib have put a nice solution in place for putting Timeouts in place. I put mine at 40 seconds, as this seems to be a good compromise between slow network speeds and users waiting forever – your mileage may vary so test this until you find a happy medium.

Summary

Overall there are a number of little things that can add polish to your app – often your users may have a totally different experience when using your app in the wild, and the closer we can get to covering all of these things to better – and happy users rate you app highly, and you get more downloads. Either way you look at this it’s a good thing.

Do you have any little niggling things you wished you’d kept in mind when first writing your Windows Phone 7 app? I’d love to hear in the comments section below.

Make Your Own Wi-Fi Hotspot - Testing Development Websites on Mobiles and Tablets

4/15/2012 2:50:27 AM

Often you need to test a website on an tablet device such as an iPad using a local development machine’s web server. For whatever reason the available Wi-Fi when developing your site may be on another subnet or network entirely to you development machine (such as in an office environment). Situations like these call for a bit of creative thinking and a different approach, so if this is a problem you face here’s my take on a possible solution.

My workplace has pretty strict networking arrangements in place as we work with Banks and government agencies. This is all well and good until you need to develop something for a mobile device and there's no way to connect a smart phone or tablet to your development machine over the network/Wi-Fi. For the guys on our team this is caused by the network subnet that is served over our office Wi-Fi being placed in a DMZ without network access (we don't want our Reddit surfing clients in reception having access to our super secret stuff).

Obviously this causes my team and I a few productivity roadblocks.

Working with mobile and tablet websites carry there own set of problems and there is a range of tools available to solve many of these. Some involve device centric visual elements such as CSS (Adobe Shadow) and user agent detection (user agent switcher). Coming from a server side development angle I've also blogged before on how to sniff mobile device traffic for debugging web service traffic on mobile devices. However none of these solutions help solve the largest problem in productivity when it comes to locally developing and testing sites from a developers machine - and if you happen to work in a restricted environment like me and my team then the pain from this problem is a lot stronger.

With this in mind, speeding up your productivity in this areas leads to big gains. If you can avoid having to deploy your development site over and over just to test something small, then you’ve smashed it out of the park.

Don't have Wi-Fi? Now you do!

The problem when boiled down is that you need to give the mobile device you are testing on wireless network access to your development machine’s web server. For people developing ASP.net websites this usually means IIS.

Now a lot of laptops and desktops these days come bundled with a wireless card as standard (this should give you an idea where I’m going with this), but up until Windows 7, Microsoft's networking stack hasn't really had the flexibility to do black magic with wireless cards in the ways that our Linux and OSX cousins have been able to. With the release of Windows 7 this changed - and with this a light bulb went off in the head of one of the developers now working for Connectify.

Connectify

Connectify is a piece of software that enables a Windows PC with a Wi-Fi device to act as a wireless access point. This enables users to share an internet connection between multiple computing devices without the need for a separate physical access point or router.

This has a whole range of benefits (maybe you are in a hotel using their internet and want your phone or tablet to get access?) but what I'll focus on here is using Connectify to test locally while developing websites.

It is extremely easy to setup, so lets take a look.

What you’ll need

A Windows 7 PC (assuming this is a given if you’ve read this far).
A Wi-Fi Card.
IIS Installed.
Your Visual Studio project setup to use IIS for testing.
Logged in as a user that has local admin rights.

Setting it up

I will use the Empty ASP.Net MVC Project template for this walkthrough simply for familiarities sake; this should work with any site you can run locally. Even ones hosted on another web server like *shock, horror* Apache or NGINX.

Firstly as housekeeping, you’ll need to setup a local IIS website that:

Accepts incoming requests on TCP Port 80.
Serves on all IP addresses (once Connectify is setup, you can get it to simply work on Connectify’s interface/IP Address).
Points at your website’s root directory (For ASP.net web sites this means the root of your website project.)

If you don’t want to install full blown IIS, you can get away with IIS Express instead.

First, download Connectify from here.

Run the installer.

When done you’ll be greeted by this screen, simply click continue.

You’ll then be asked whether you want to use the “Lite” or “Pro” version of Connectify. Select Lite as this is all we’ll need.

This will leave you with the panel on the right of the screen asking you to setup your HotSpot.

Enter a HotSpot name (the Lite version requires that the name starts with “Connectify-“), password (must be 8 characters or over), and then select the internet connection you would like to share.

Even if you don’t work in an environment where security is considered “critical”, make sure that you set a REALLY strong password. After all, you are setting up an additional way for people to get onto your network so this is a pretty big deal in an office environment. Check out Troy Hunt’s great post on passwords for easy ways to manage these.

For me, I am already on Wi-Fi as I'm writing this post on my laptop, but you may want to select a different connection to share – the interface you share doesn’t matter too much as you’ll only be accessing your local IIS installation; as long as you don’t have firewall rules setup on the interface you’ve selected stopping port 80 access you should be fine.

Then hit the “Start HotSpot” button.

Now on your mobile device, connect to your newly configured Wireless Access Point by changing the settings on your device’s Wi-Fi connection.

Now on your host machine, open a command prompt window and type “ipconfig” to pull up the list of interfaces on your machine. You should notice that you now have an additional Wi-Fi adapter – this is the virtual one created by Connectify to serve your devices. Take note of the IP Address listed, as this will be the address your site is hosted on.

Now enter this address in your device’s browser proceeded by “http://”. This is the IP address of your new virtual Wi-Fi adapter (mine was http://192.168.96.1).

If you have trouble getting this working (your device maybe sits there waiting for the page to load forever), then Windows Firewall or similar may be blocking the request. You can add IIS to the Allowed List by going to “Settings > System and Security > Windows Firewall > Allowed Programs” and ticking the box to allow IIS (World Wide Web Services (HTTP).

Special Note

When you have finished testing your local website, it is important to click the Stop Hotspot button in Connectify to turn off your newly created Wi-Fi access point. This will ensure that no evil doers can run any dictionary or rainbow table attacks on your machine while you aren’t expecting it (probably while you are at home over the weekend or overnight).

Summary

There are now a number of tools available to assist with testing sites on mobiles and tablets, but few tackle the problem at its source as well as this does: speeding up the rinse repeat of a developer test-deploy-modify cycle. Hopefully I've shown you a way to drastically reduce the time spent on this when working locally on your developer machine.

What tools do you use to help speed up mobile website development? I’d love to hear in the comments below.

Cause for Concern - Piracy on Windows Phone 7

4/25/2012 5:51:54 AM

I have been playing with the Windows Phone 7 SDK for a while now, however I have been lucky to still have a day-job while doing my tinkering and therefore haven’t sourced my main income from sales in the WP7 marketplace. There are others who don’t have the same luxury as me and have bet a considerable amount of their time on the platform to date. Whether these developers are aware of it or not they are fighting a silent battle that I want to bring more awareness to – a problem that every smart phone ecosystem has faced to date: Piracy.

This problem is a worry as it affects Windows Phone devices more than some other platforms as we write our applications in languages that can easily be decompiled: get an app’s binaries, and you have mapped the apps very genome, its source code. A problem which once done is out there forever – if people get your app’s code or resources now, the game is up even if Microsoft releases a fix in the future. Hopefully by raising more awareness of this, together we can challenge Microsoft to put more effort into this area moving forward.

NOTE:
I need to be really clear before I start. This post does not promote piracy on the Windows Phone 7 platform. I personally develop apps for Windows Phone 7 and have a vested interest in the platform’s success. Hiding facts that are publically available does not increase this behaviour but it definitely brings it out of the shadows for all to see (and implement change).

What I am trying to do is bring up a topic that is not a new one for Windows Phone but appears to have been un-addressed by Microsoft since 2010. This issue can’t go away until more effort is made in this area or the problem will only get worse as time goes on.

On a side note, I also tried to contact a number of people at Microsoft to get their view before posting this post however, after a week or trying, I am yet to hear from anyone.

What this post will cover:

Piracy on Windows Phone 7: It exists, and more attention needs to be given to it.
How to pirate a Windows Phone 7 app.
A closer look at how the piracy app I'm using works.
What can you as a developer do about this problem on your apps.

Piracy on Windows Phone 7

All of the smartphone “walled gardens” share one thing: Piracy. Apple being the leader in this space with the iOS platform and knows this better than any of us; People have been pirating iPhone games for years with some stats showing that for every app sold 10 more are pirated on the IOS platform.

Windows phone 7 has been no different, with WP Central reporting that piracy was easily achievable way back in 2010. There have been many such reports since then. There is still little or no blocks in someone’s way from firstly side loading apps easily onto devices (be they pirated or not) or more worryingly simply sniffing the Zune traffic and then downloading the XAP directly from Microsoft. It’s now nearly 2 years since this news first came to light and little appears to have changed; there are still apps out there that enable these same hacks to be conducted easily by the everyday lay person.

My large concern about this is the double whammy that is Piracy on WP7 - iOS apps are built in objective-c, which while not making iOS apps immune to decompilation; it does make the act of decompiling a lot more challenging for any would be pirates.

Windows Phone 7 apps don’t have anywhere near this level of complexity when it comes to decompiling the source code and assets that a game or app is made with. We build our apps using the .Net framework in any of its supported languages, which in turn gets compiled into an intermediate language (MSIL) that by its very nature is powerful because it’s not low level.

As I'm sure you are all aware, If you get a .Net framework assembly (.dll) and arm yourself with on of the multitude of decompilers on the open web, you very soon have a readable copy of a binary’s source code.

If you are a Windows Phone 7 developer this is all a little unnerving when you consider how much time and effort you’ve poured into your app.

Taking a closer look at the facts:

Device owners can easily unlock their phones for side loading fun both legally and using hacks (even if semi legitimately done with something like ChevronWP7).
Getting an unadulterated copy of an application or game’s XAP file is trivial (more on this in a sec).
Once anyone has the XAP for your app, they can then easily grab your source code and resources, further allowing them to do what they want with it. Side load it for free, or extract your app’s resources for nefarious purposes.

My concern grows when I consider that in the last year there have been instances where pirates have downloaded an app’s XAP from the Windows Phone 7 marketplace, created a developer account and submitted the same app themselves (with Microsoft approving it). There has also been a case where pirates have downloaded an App, copied its contents and created a competing app with the original developers resources (in this case their Yoga video content, created by their wife…).

There is little we can do about the first or third step mentioned above, but I think Microsoft can do more about the second item – the ease of getting an apps package.

Is this really a problem?

Over the last few days I have been touching base with a few people in the community about this. Many have said similar things: prove this is a problem, and we’ll give it more thought.

One comment from Justin Angel (a Nokia principal engineer on WP7) said this:

While I must apologise before I go on in reply to Justin’s comment, as I’m not beating up on him personally, the view he mentions on piracy as “Prove it’s a problem” for a relatively new platform like Windows Phone is dead wrong.

If as developers we pour part of our lives into creating apps for a smart phone platform, we are often choosing to do so over another. If there is a feeling that the creators of your chosen platform care little for the security of your intellectual property, that’s a pretty good reason to not bother with it. This is especially the case with Windows Phone where it is so easy to reverse engineer an app to retrieve the app’s IP.

Justin’s stance does nothing to support the developers choice for Windows Phone 7 or allay any fears of current developers on the platform.

If I can download my app’s XAP and open it as a zip file this is what I see:

if I then decompile one of these binaries this is what I see… real code (if you haven’t obfuscated your apps code):

namespace InTheKnow.PhoneApplication
{
  public class MainViewModel : INotifyPropertyChanged
  {
    private PropertyChangedEventHandler PropertyChanged;

    public ObservableCollection<GoogleAnalyticsAccount> Accounts { get; private set; }

    public bool IsDataLoaded { get; set; }

    public bool IsFirstTimeLoad { get; set; }
  }

This means that anyone getting my app potentially has my IP – whether the numbers of piracy are high today or not, this is a problem.

What the bad guys keep on the down low

If you do a Google search looking for Windows Phone 7 pirating, you’ll soon come across a few Windows desktop apps floating around that make the act trivial. I will not name them or link to them, but I will walk you through using one of them to download my personally developed Google Analytics app InTheKnow.

I have downloaded an app I found that touts itself as being a one stop shop for Windows Phone 7 pirating. It downloads, removes security and side loads all in one.

I have blocked out any branding from the screenshot below to make it harder for budding pirates reading this post to find it, but you can see from its interface that when I search for my app that it offers a similar experience to that of the Zune Marketplace client (I can only assume it uses the same web service endpoints).

Once I’ve search for my app and found the version that I want, a simple double-click is all that is needed to download it.

I decided at this point to do a little digging to see what the creators of this app were doing that was magical enough to get around my perception of the security Microsoft would have put in place for the marketplace.

Taking a look at this user journey using Fiddler, I became quite concerned.

None of the traffic coming or going was over HTTPS, and none of it consisted of anything but plain vanilla Http GET calls.

The app queries the Zune store using a search URL, which in turn returns a GUID along with all the store details for an app.

It then queries another URL using the returned app GUID, which returns XML containing the download URL for the app’s package.

If you copy and paste this last raw URL into a freshly started web browser (i.e. no session, no special headers), this downloads without hesitation. It’s just a plain old URL – you could email it or share it just the same.

The XAP you download only has one protection to it – an XML file that contains a signing signature. Something that you can find an easy way around with the help of another Google search – hell, the marketplace piracy tool I was using to write this post even would patch this for you upon download! How convenient…! *sarcasm*

Again to reiterate, all of the simple things Microsoft could have done:

No authentication occurs.

SSL is not used.

No special headers or HTTP tricks are used.

And that was that. I now had the XAP file for my app, without paying. And as only a cursory Google search will tell you, XAP’s aren’t hard to get the contents of – they are just zip files.

But you have to put it on your phone still, right?

At this point regardless of the fact that I now have the binaries and resources for my app (i.e everything there is to my app), if I was to side load this app on my phone I’d still need to have a developer registered phone, so that’s a hurdle right?

Wrong – the application I was using kindly came with a XAP uploader as well that works with non-developer unlocked devices. Two more clicks and I was done.

This sucks! What do I do to protect my apps

So Microsoft might not have the answers yet to the piracy problem, but there are things you can do to attempt to minimise the damage.

Obfuscate your code

Microsoft has teamed up with PreEmptive Solutions to package the Community Edition of their code obfuscation tool “DotFuscator” with Visual Studio. Pre-emptive has also released a full blown Windows Phone Version that is also free and can be downloaded from here that includes support for Silverlight XAML obfuscation as well.

These tools obfuscate the code in your binaries, so that once the bad guys get your XAP file they will find it harder to understand any code they decompile – any resources such as videos or images will still remain free to copy or edit though so keep this in mind.

It is worth noting that some obfuscation tools often can and do slow your app down.

Example of your code before Obfuscation:

private void CalcPayroll(SpecialList employeeGroup) {
    while (employeeGroup.HasMore()) {
        var employee = employeeGroup.GetNext(true);
        employee.UpdateSalary();
        DistributeCheck(employee);
    }
}

Example of your code after using Dotfuscator:

private void a(a b)
{
    while (b.a())
    {
        a = b.a(true);
        a.a();
        a(a);
    }
}

My advice: Obfuscate every app you submit to the marketplace.

Code checks

Some developers are using code that looks for the presence of DRM files in the app’s directory and if missing, launches the marketplace to purchase.

public static bool IsHacked()
{
    try
    {
        #if DEBUG
                return false;
        #endif
 
        //scramble WMAppPRHeader.xml file name to make life a little harder in case of reverse engineering
        string fl = "xxxWxxxxMxxxxAxxxxpxxxpxxxPxRxxxxxHxxxxxxxexxxxxxaxxxxdxxxxxxxxerxxxxx";
        fl = fl.Replace("x", string.Empty) + ".xml";
        XDocument doc = XDocument.Load(fl); //is hacked, this file is missing or empty!!!
        return false;
    }
    catch (Exception)
    {
        MessageBox.Show("This app was pirated and is not safe to use, please download the original one from Marketplace.");
        var marketplaceDetailTask = new MarketplaceDetailTask
                                        {
                                            ContentIdentifier =
                                                PhoneHelper.GetAppAttribute("ProductID").Replace("{", string.Empty).
                                                Replace("}", string.Empty).Trim(),
                                            ContentType = MarketplaceContentType.Applications
                                        };
        //ProcutdID will be changed after AppHub certification, so has to be read from manifest!

        //download Coding4Fun toolkit for this helper
        marketplaceDetailTask.Show();
  
        return true;
     }
}

You could consider the implementation of these techniques, but be warned: This can be touch and go as you risk making your application non-functional if Microsoft changes their DRM functionality or something out of the ordinary happens in the legitimate deployment of your app.

Summary

We cannot stick our heads in the sand any longer – if you are a member of the Windows Phone 7 team and are reading this, please help me raise this with your team members. I love the platform and want it to continue, but as a developer I need assurances that these things will get attention – not take years to fix as they appear to have to date.

A web service similar to the one suggested here, would be a great active validation technique that should limit the amount of effort the WP7 team put into it.

If Microsoft wants developers to invest time and money in creating apps for their platform, they have to have a good story when it comes to Piracy – currently this doesn’t appear to be the case.

Come One, Come All to DDD Sydney – June 30th

5/12/2012 9:38:13 PM

I’ve written about local conferences a few times before, but DDD Sydney is one of my favourites. There are few conferences that are so “For Us By Us” as DDD Sydney as it’s organised by Lewis Benge and contributed to by a whole swath of the local developer community from a number of different user groups, so it’s a great place to come down and meet a number of your local devs, learn something new or take part in the discussions – and for $25 it’s one of the cheapest conference tickets around!

Details:

http://www.dddsydney.com/ (ONLY AU$25!!!!)

Agenda being voted on now:

http://www.dddsydney.com/Voting.aspx

A number of fellow bloggers from the Sydney scene will being there, so remember to stop by an say hi (Troy Hunt, Richard Banks, Mohamed Meligy, and many others).

Vote for my talk!

This year is special for me at DDD, as I have submitted a talk to DDD for the first time (go here and vote for me, please!).

My talk is titled:

A few things Developer's should know about the Internet (But probably don't)(Level: 200 )

Doug Rathbone teaches us that although we can search high and low for the next big thing in programming, often knowing a bit more about how our apps and websites talk on the internet pays dividends. Come take a look at some tips on using TCP/IP, DNS, and Email that will give your next project the edge.

I’m basically going to take you on a journey of a range of points from my posts about TCP/IP, Email Deliverability and DNS as well as how it all fits together. My inspiration for this has come from my travels in our industry and the realisation that a lot of us miss a lot of knowledge about what goes on behind the veil of the internet. Hopefully you’ll learn a little bit about how the internet works so that you get home earlier, and your clients are happier.

I can 1000%* Guarantee that you will love my talk, so tell your friends and family to vote for it!