Diary of a Ninja

Show Your Recipients You Care – Avoid noreply@

7/11/2012 6:08:47 AM

When delivering messages to people using email, companies and website owners have fallen into a common fallacy about the internet: believing its OK to show contempt for our readers by not caring for their reply. We do this every time we send an email, however important, that comes from noreply@mywebsite.com. Like a number of life’s oddities this doesn’t make sense – let’s look at why and how we can change it.

As old as time Itself

Sending email to your visitors, customers or anyone that is important to us is an extremely important task. E-mail is the modern day equivalent of sending a letter – at its basic level; human communication. Not too long ago, people used to spend inordinate amounts of time penning letters to each other with important information, things to watch out for, or good news. You name it, someone would have probably made the decision that taking the time to send a letter or message to another human being was worth the effort.

And guess what?

Not much has changed.

Humans still feel the need to write each other messages. Entire cottage industries (is Twitter considered a cottage industry?) have developed for differing levels of modern communication.

And yet when it comes to e-mail, companies and website owners seem to be show contempt for their message’s recipients in what can only be called the status quo of modern broadcast communication.

We send them a message from an email address that cannot be replied to.

We make it clear we don’t care for their response.

Why this is bad #1 – Emotional

Situate yourself as the recipient;

As a sender, if you can’t show that you’re bothered to read my response then why should I give you the time to read your email?

What if I want to give you feedback? What if I want to tell you you’re awesome? If you are a start up this is the best thing you can ever receive, why would you want to ignore it?

Getting a response from your subscribers, customers or visitors means they care.

Either they care enough to spend their time abusing you. Or spend their time telling you what you could change. Or spend their time telling you how much they appreciate your product or service.

All good things – why miss out?

Why this is bad #2 – The Dreaded Spam Filter

A number of spam filters use the words noreply or dontreply as trigger words and therefore give your e-mail a higher spam score. Do you want your email even delivered?

When people reply to your e-mail, many spam filters and webmail services (Gmail etc) put your e-mail address on a whitelist.

This ensures that you don’t ever get flagged as spam – you get this awesomeness for free just by making yourself available for reply. Why would you want to miss out on this?

The American CAN-SPAM act gives us a legal reason for not doing this;

The CAN-SPAM Act: A Compliance Guide for Business

“Your “From,” “To,” “Reply-To,” and routing information – including the originating domain name and email address – must be accurate and identify the person or business who initiated the message.”

In America and Europe MAAWG give us further guidance;

MAAWG Sender Best Communications Practices:

“Senders should have the capability to process email-based unsubscribe requests. Senders should also consider making offline unsubscribe mechanisms available. The sender’s ‘From’ or ‘Reply-to’ email address should also be able to receive unsubscribe requests, unless otherwise indicated.”

What we can do

When building your next website, or firing up that next Campaign Monitor email blast, use a real email address; one that has a mailbox attached to it that a human reads now and then.

Some options:

support@yourwebsite.com
info@yourwebsite.com

And before we get all worried that the mailbox we’re sending from being filled to the brim with crap consider a few rules you can implement on that mailbox to keep you sane:

Weed out the “postmasters”, “delivery failure” and “delayed” messages with mailbox rules.
Send using a lesser known part of the SMTP RFC; + addressing to separate out different mail from different sources using simple rules.

Summary

Sending email to your customers or visitors for marketing, usability (sign up validation) or just to say hello can be a great way to keep them engaged and foster two-way communication. If you’re using a noreply@ address as your sender, your conversations will always be one-sided. You’re missing out.

If you are using a noreply@ address, it’s time to change. Why not do it in the open? Send an email informing your customers or visitors that you will be changing your address shortly and encourage them to add the new one to their Safe list.

Tell them you welcome their feedback with open arms and show them you care – it can only be a good thing!

Dear Microsoft, Please Include Web Deployment Projects in Visual Studio 2012

7/15/2012 5:27:32 AM

Microsoft Web Deployment projects are an easy way to add a MSBUILD scripting to your Visual Studio web projects. I use them all the time for personal deployment projects and at work so do all my team members. With the upcoming release of Visual Studio 2012 there is currently no Web Deployment project type. Luckily there is something we can try and do about it – Let Microsoft know.

If you hadn’t noticed, I love Continuous Integration. CI makes up so much of an important part of my development flow that I don’t deploy any other way anymore. It’s a matter of religion for me.

My personal projects are deployed through TeamCity, Web Deployment projects and Git.

My works projects are deployed through TeamCity, Web Deployment projects and SVN.

Some of my private clients are deployed through Bamboo, Web Deployment projects and Mercurial.

I simply love Web Deployment projects and the power they make available to you – MSDeploy is great, but is a very specialised tool for deploying over Http.

Web Deployment projects on the other hand allow you to script anything using MSBUILD scripting and integrate it into your build process.

Visual Studio 2012 Web Deployment Projects – An Unknown

In Visual Studio 2010 Web Deployment projects were not included in the standard install. They were available as an aftermarket download.

Currently the Visual Studio 2012 RC doesn’t have the Web Deployment project type included either.

What is more unsure is whether Microsoft are actually going to build the add-on anytime at all.

There is the possibility that if they don’t think it is used by many developers that Microsoft simply won’t be creating one. (!!!!!)

Have your say!

If you visit the link below to vote for Web Deployment Projects to be included in the new version of Visual Studio 2012.

Vote for Web Deployment projects!

Autoplay, a New Type of Popup Plaguing the Web

7/10/2012 7:22:42 AM

When placing audio and video elements on a web page I’ve worked on a number of pieces of work where, for one reason or another, clients want to have their media Autoplay. To us nerds it may seem like common sense that Automatically playing media to a visitor is a bad idea for accessibility. The W3C has made this clear with it’s WCAG guidelines – we’re nerds; we care about these kinds of things. It’s worth mentioning that as with most accessibility features though, proper use of Autoplay also does a lot for usability and visitor sanity for the rest of your audience as well.

In the last 6 months I’ve had a number of projects that have potentially involved autoplaying video or audio to visitors. Some of these pieces of work have been full blown interactive arty web sites where the target audience is expecting it (similar to The Wilderness Down Town), while others have simply been over excited clients or creative/designer folk wanting to make “something epic on the interweb”.

Autoplay is not a new thing, however like a lot of things that aren’t necessarily “the right thing to do” on the web, there are still a number of brands and companies doing it that should really know better.

“Crap!… mute, mute mute!”

If you’ve ever been rocking out to some tunes and visited a site that autoplays some media content with the volume on full you’ll know first hand how annoying this can be.

If this happens at work and that video you’ve just clicked on of an Asian German man techno yodelling with chickens just autoplayed, you might find yourself attracting the wrong kind of attention to your work browsing habits – YouTube makes this very clear by getting this wrong; they autoplay. My work colleagues have been wondering ever since…

Picture this same scenario but instead put yourself in the shoes of a blind website visitor – if a website is playing a video using a flash video player and a visitor viewing the site is using a screen reader, the flash video player becomes for all intents and purposes “invisible”. That pause button the website’s flash developer has placed on the page to stop the video also becomes “invisible” – so if you autoplay video, they may potentially be unable to stop the video playing at all.

Now if only the developers working on The Sydney Morning Herald understood this!

People who do this stuff for a living – the W3C

When it comes to building websites for a client that mandates accessibility such as the Government or a major corporation the main source of reference that everyone turns too is the W3C and their Web Content accessibility guidelines (WCAG 2.0).

In the section of the specification talking about implementing audio, there is a quote that is definitely worth a read (Understanding Success Criterion 1.4.2 [Audio Control]) (source)

“…
Note: Playing audio automatically when landing on a page may affect a screen reader user’s ability to find the mechanism to stop it because they navigate by listening and automatically started sounds might interfere with that navigation. Therefore, we discourage the practice of automatically starting sounds (especially if they last more than 3 seconds), and encourage that the sound be started by an action initiated by the user after they reach the page, rather than requiring that the sound be stopped by an action of the user after they land on the page.
…”

If you are implementing anything that allows for visitors to Pause, Stop and Hide multimedia they also mention the following (source):

“…
Moving content can also be a severe distraction for some people. Certain groups, particularly those with attention deficit disorders, find blinking content distracting, making it difficult for them to concentrate on other parts of the Web page. Five seconds was chosen because it is long enough to get a user’s attention, but not so long that a user cannot wait out the distraction if necessary to use the page.
…”

The future of media on the web – HTML5 Audio/Video

The current work-in-progress HTML5 specification from the W3C includes new attributes that support autoplaying media with the autoplay attribute of video and audio elements with the following statements appearing inside the “HTML living standard”:

“…
Authors are urged to use the autoplay attribute rather than using script to trigger automatic playback, as this allows the user to override the automatic playback when it is not desired, e.g. when using a screen reader.
…”

Many agree that having this option included in the specification is a good thing as Simon Pieters points out:

“…
Removing the attribute will not make pages stop autoplaying video. Instead they will use script to make videos autoplay, and then it becomes harder for the user to prevent videos from autoplaying. (You could have a pref in the UA to disable autoplay.)…”

A word on walled Gardens

Whether you love or hate the impact that Apple mobile handsets have had on the market, you’ve got to give it them that sometimes their opinionated take on how they implement or not implement certain functionality sometimes is spot on.

Emma Sax mentions in her post that Apple has made the decision to disable autoplay on iOS devices, both using script blocking techniques and removal of support for the attribute implementation (source):

“…
In Safari on iOS (for all devices, including iPad), where the user may be on a cellular network and be charged per data unit, preload and autoplay are disabled. No data is loaded until the user initiates it.
…

This plays the movie: <input type="button" value="Play" onClick="document.myMovie.play()">

This does nothing on iOS: <body onLoad="document.myMovie.play()">
…”

They also mention in the same document

“…
It is especially important to provide user controls on iPad because autoplay is disabled to prevent unsolicited cellular download.
…”

From recent tests we’ve done for clients on a number of current Android devices (Galaxy S3, HTC Desire) it appears that many mobile devices appear to ignore the autoplay attribute all together. Based on this I'd hesitate to even consider the autoplay attribute “usable” in a commercial development sense based on the ever growing amount of mobile traffic from visitors.

Advice if you have to “go there”

If you are working on a site that has a really strong need (subjective, or actual) to include Autoplay there are a few things you can do to make your site less painful for your visitor;

Autoplaying your media is a lot more acceptable if your intent to play media was made before a visitor came to your page i.e. following a link that says “Click here to view this video”.
If you ignore all I've mentioned in this post and use autoplay, ensure that your clip only lasts a few seconds to ensure that you are also abiding by the W3C spec when it mentions how you should build ‘Moving, blinking, scrolling’ content.
If using video, autoplay with the volume muted, and give users the option of “unmuting” your media. This will stop non-vision impaired users from being embarrassed while ensuring the vision impaired can still navigate your site.
If the clip lasts more than a few seconds, make sure that you include an extremely visible, easy to find way for your visitors to pause or stop your media. This needs to be done using semantic HTML mark-up – not just a larger “invisible” button in your flash video player.

Stop! Thief!

The motivation for this post was like many things in my life inspired by frustrations I had experienced during my work day. Like others may have done when arguing a point with others that weren’t close to the subject, I searched for reference material to back up my argument at the time as to why we “shouldn’t use Autoplay”. One of these places of reference was Emma Sax’s post on exactly the same topic. Recently I was contacted by Emma Sax on twitter about the lack of reference I had made to her post on Autoplay and her feelings that I had plagiarised some of her writings. When I wrote this post I was lazy in that a number of her quotes from the W3C as well as Apples SDK were copy-paste jobs. This was unprofessional and crude on my part but was not meant to cause harm.

I have since rewritten a large part of this post to further separate her opinions from mine and apologised to Emma. We both share similar feelings on two things: Autoplay being bad for the internet; and due credit being given where it’s due. Emma’s post is definitely worthy of credit.

Micro Benchmarking Your ASP.Net Pages Using Apache Bench

7/30/2012 8:05:45 AM

In ASP.net land we are often lead to think the “Microsoft way” when it comes to a lot of things. Running performance tests and benchmarking is one of these tasks where we are often found looking into commercial tooling or products to help us find out how our applications handle load. Meanwhile a lot of web developers on other stacks are doing it with great free tooling. There is nothing stopping us from stealing the best parts from these stacks and bringing them back to the land of ASP.Net.

Image: Martin Heigan

Tooling tooling everywhere…

If you’re developing ASP.Net websites and web services you’ve probably turned to a few different options to performance test your pages and web services.

You may be lucky and happen to own a copy of Visual Studio Ultimate and have had a play with the Performance and Stress testing tooling that comes along with it.

In the non-microsoft tooling world I’ve used a few tools such as Load UI and Soap UI for a number of projects successfully. There are many many commercial products that do similar jobs.

But a lot of these tools sometimes actually don’t make it really clear how to get answers to some simple questions that both we and our clients often want to know.

Simple questions like:

“How many users can my application take at once?”

Or other gems that you might like to know;

“How long does my page take to execute? How long does it take when 100 visitors are using it at the same time?”

Info for nothing, and the tests for free…

When you install Apache you get a whole heap of tools along with it. We normally think of Apache as a Linux/Unix web server, but there is a windows version that contains all the same power.

One of the included tools that comes with Apache is the performance testing tool Apache Bench.

Apache Bench is a really powerful and simple command line tool for performance testing and gathering metrics on how your application pages respond under load. It does this with very little CPU usage and a very small memory footprint, allowing you to run it on hardware that has lower spec than a lot of the commercial tools around. This makes it great if you want to use old machines on your network to run testing concurrently form different network addresses.

Apache Bench allows you to simulate a defined number of requests at a time for a set amount of total requests and view results on how your pages respond.

This allows us to do the following:

Test a page with 10 users at a time for 1,000 requests
Test a page with 50 users at a time for 10,000 requests
Test a page with 100 users at a time for 100,000 requests
etc.
etc.

Do the above for 1 minutes, 5 minutes…

And get response times; failure rates; requests a second; transfer rates for each level of user load.

Actually get data that can help you estimate how your application will function once it goes live.

For Free.

WARNING
This is not a “this is how you should perform tests on your website to become invincible”. There are plenty of blog posts and articles out there to get you strategies an approaches on how to do this. This post is however a guide to doing basic micro-benchmarking on individual pages during development to test performance in ways that you won’t be able to do on your dev machine without other software.

Performance testing and website load generators such as Apache Bench can be dangerous things in shared hosting environments and without your webhosts knowledge, so please take precautions before testing on other peoples sites or hardware.

Performance testing - Thinking it through

As web applications use networking bandwidth, you need to take this into account when deciding how you are going to be doing your testing.

Are you testing just your application? or are you testing your application and your server’s internet bandwidth? (and your test machine’s bandwidth…)

If you want to know how your application responds independent of the internet, you should test from a machine that is on the same internal network as your webserver to ensure that your internet pipe is not the bottleneck in testing. This will give you upper bounds on your application, but will not give you the full picture for launch day.

If you want to also test the internet link that your server is serving through (i.e. the real-world), run your tests from machines outside your server’s internal network – somewhere on the internet.

Maybe an Amazon EC2 Instance? (If so use the Linux Apache Binaries – Linux instances of EC2 are cheaper…)

I do suggest you do this with your ISP’s knowledge and only when you are running dedicated servers – otherwise you may become a willing participant in a DOS attack against your web host, and also take the other sites on your box down at the same time from the performance hit.

Making it all happen

Download the XAMPP zip file.

In the bin folder of the zip file extract the file ab.exe into a location on the machine you will be running your tests from so that you can easily get to it.

For the point of this demo I've placed mine in “C:\testing\”.

On the machine you will be testing from open a command prompt window and move into the above created directory (enter “cd C:\testing\”).

Now we will start by issuing 1,000 requests from 5 concurrent clients at the same time.

The –n is the number of requests to process (we want this to be a high enough number that the tests run for a minimum amount of time (a minute or 30 seconds should give you an idea on numbers)

The –c is the amount of concurrent requests to run. This is the equivalent of a number of people hitting your site at the same time.

ab.exe -n 1000 -c 10 -b http://mywebsite.com/mycontroller/myaction

This will return a result similar to the below ( the below result was run with 10,000 requests from 50 clients against my blog home page):

As you can see from my results above my page could handle requests at a rate of 105.90 requests per second, the fastest request was 422ms, the slowest 4307ms. While I have pummelled this page - I'm making 50 requests at once over many requests, these response times are simply not what I had hoped.

My testing above was conducted with Apache Bench and I purposely made this page perform worse by turning off ASP.Net caching. While I am not testing anything but my home page this does show me that my homepage needs work. I want to get the homepage response time down to around the 100ms mark.

If I turn my ASP.Net MVC controller caching back on this result then becomes much better:

This is now handling requests at a rate of 871.64 requests per second, the fastest request was 57ms, the slowest 180ms. Wow – that’s a lot faster.

When testing your pages you want to start with a number of concurrent sessions where the response times and requests per second are repeatable and fast (around ~100ms), and then walk this number up until you get a feel for the peak your page can take.

So try with 20 concurrent requests

ab.exe -n 1000 -c 20 -b http://mywebsite.com/mycontroller/myaction

If this is successful while still keeping ~100ms response times try 30, and then 40 and then 50 concurrent requests (the –c 20 is the number of concurrent requests) and so on and so forth until you start to see you application degrade (degrade = have a higher response times than say 300-400 ms).

Taking it one step further

So now that you’ve micro benchmarked a single page it’s time to think about ways we can take Apache Bench further. We want to test multiple pages concurrently – maybe with differing loads (as not everyone will be surfing your site in an even spread across all your pages.

Batching requests.

When people surf your site during a busy period you will have differing loads across differing pages and your testing your reflect that.

As Apache Bench is a command line application you can easily batch multiple executions together to create this load.

As it supports command line trunking we can save results out to text files.

If I want to have say 50 users on the site at once and I look at the split of numbers I’m wanting to test above, I can easily spread this across multiple concurrent Apache Bench sessions from my test machine.

Place something similar to the below in a batch file to simulate different areas of your site being accessed at the same time.

REM 30 users accessing a content page
start cmd /c ab.exe -n 1000 -c 30 http://mywebsite.com/mycontentpage > results-contentpage.txt

REM 10 users accessing a forum page
start cmd /c ab.exe -n 1000 -c 10 http://mywebsite.com/myforum/thread-123 > results-forumpage.txt

REM 7 users accessing the homepage at once
start cmd /c ab.exe -n 1000 -c 7 http://mywebsite.com/ > results-homepage.txt

REM 3 users accessing the support section
start cmd /c ab.exe -n 1000 -c 3 http://mywebsite.com/support/list > results-support.txt

Save this as a new batch file apache-bench-tests.bat and run it from the command line – this should spawn a number of command line sessions concurrently all running Apache Bench.

The results from each of your bench marks will be saved in the text files mentioned after the > in each batch run.

Testing from multiple machines at the same time.

If you’re wanting to really test your application in a more realistic setting you need to test everything – including your network hardware.

For this I would suggest setting up a collection of machines or virtual machines and run batch scripting on each of them with Apache Bench to simulate your site from multiple physical locations.

Testing forms and search pages by posting data.

You can test your application using more than just GET requests.

Apache Bench supports http PUT and POST and can simply take a text file with your POST or PUT data in the command execution.

ab.exe -n 1000 -c 5 http://mywebsite.com/ -p file-that-you-will-post.txt

ab.exe -n 1000 -c 5 http://mywebsite.com/ -u file-that-you-will-put.txt

Summary

The above is not a scientific in depth approach to performance testing, but if it will definitely give you a better feel for how each page of your application responds under load and on the cheap.

By scripting multiple pages of your site to be hit concurrently you will get a clearer indication of how it will perform in the wild and which pages may need optimisation or caching to ensure your site can handle extra traffic concurrently so that when those spikes happen on launch day, you will either be better prepared or know how bad the situation will be – Profiling and performance work should get you the rest of the way there.

Up Log Creek Without a Paddle – Part 1: Windows Audit Logs

8/15/2012 7:14:33 AM

When bad things happen to either your website or your server you’re usually faced with a situation that either makes or breaks you. Much like having a good backup and restore plan, being able to filter and scan log files for what you need to help draw conclusions on how a situation occurred or by whom it was conducted, is an important part of your security plan. However if you have a heavily traffic’d website, network share or part of your file system and you’re doing a lot of logging, you probably have files the size of the moon to wade through, so making sense of them can be a nightmare.

This is part 1 of 2 in a series on how to wade through the information overload that can come when searching giant logs files.

Photo from slack12

Searching through Windows Auditing Logs

If you have Windows File System Auditing turned on, you should be able to see all sorts of valuable information about actions that have occurred on your file system.

Files being created, saved, deleted (maybe by your IIS user?).
- has a hack occurred that has provided the IIS user with more access than wanted?
- has someone discovered a security hole in your site allowing the IIS user to do things unintended?
Web.config files being changed (Either by the IIS user, or an admin user).
Has a third party user gained access to your server and logged in by RDP? Has your webhost done so without your knowledge?

While IIS logs can help with shed light on some of these things, actual file level actions are best recorded using Windows File System Auditing as this can be setup to record not just logs of requests such as those recorded in IIS logs, but also file opens and edits inside an RDP session, an FTP session and generally any other point of interaction with the file system.

This can be an important record when it comes to retracing the steps of an attack against your server – if you just remember to turn auditing on in sensitive parts of your file system!

Once you do turn it on however the next problem arises when it comes to investigating an incident on your server: Oceans of data.

So much data, that the Event Viewer itself is unable to filter and search this haystack at anything but a glacial pace.

Finding a Paddle

This problem can get really bad. I’ve had to search through Windows Event Logs that were many gigabytes before, and the event viewer simply packs it in.

So what can you use to scan logs these big?

Microsoft has the answer in the form of an unsupported tool called Log Parser 2.2 that is just the ticket.

While I'm definitely not the first to talk about this tool (Jeff Atwood was writing about Log Parser in 2005!) it’s a utility that often goes unknown outside the server admin world.

Although it was released in 2005, Log Parser 2.2 is still a very powerful and an extremely fast command line tool used to query log files using SQL queries.

When it comes to taking on the monumental task of querying 100’s of megabyte or 100’s of gigabyte log files, it soon becomes your best friend.

logparser.exe "SELECT * from 'C:\LogFiles\example_windows_eventviewer_file.evtx'" –i:EVT

2012 – Log Parsing; Now With GUI!

Log Parser 2.2 was released in 2005. The log formats for a lot of different applications haven’t changed much in this time – although a number of things have occurred in the “Log Parser world” since.

Lizard Labs has released Log Parser Lizard, a GUI wrapper for Log Parser.
The Exchange team has released Log Parser Studio another GUI wrapper for Log Parser.

I still use the command line exe to query large files. I’m not sure if its simply instinct or placebo, but it feels faster (I haven’t tested it), but I almost always use Log Parser Lizard to build my queries before running my command line log searches.

The main reason I use this is that Log Parser Lizard has intellisense to help write your queries (this saves you a lot of time)

The Log Parser Lizard application itself does have a downfall, in that it is not being written in a very async manner, meaning that if you use it on extremely large files, Log Parser Lizard will block the UI thread and hang – another reason that I use the command line to scan large files.

The Exchange team’s tool is written to not block the UI while processing files, but is very feature poor compared to Log Parser Lizard.

So my recommendation:

Build your queries using Log Parser Lizard on a smaller subset of your log file.
Copy your query into the command line and run it against the command line tool.

Example queries for Windows Security Logs

If you have file system auditing turned on, and you export your Windows Security logs to external files (evtx), you can then use Log Parser to query them directly.

For security logs, i’ve created a few sample queries, to give you an idea on some searches that help provide intelligence on what has occurred on your server and to get you started.

All actions against a folder within a certain timeframe

“What’s been happening in the folder that my website/application stores data in between date X and date Y”

logparser.exe "SELECT TimeGenerated, EventID, EventType, EventTypeName, EventCategory, EventCategoryName, SourceName, Strings, ComputerName, SID, Message  FROM 'C:\LogFiles\example_windows_eventviewer_file.evtx' WHERE Strings LIKE ‘%C:\\MyWebsiteRootFolder%’ AND TimeGenerated BETWEEN timestamp(’04/04/2012', ‘dd/MM/yyyy’) and timestamp(’06/04/2012', ‘dd/MM/yyyy’) ORDER BY TimeGenerated DESC" -i:EVT

All successful file actions

“Who/what users have accessed my folders and files inside a certain directory”

logparser.exe "SELECT TimeGenerated, EventID, EventType, EventTypeName, EventCategory, EventCategoryName, SourceName, Strings, ComputerName, SID, Message FROM 'C:\LogFiles\example_windows_eventviewer_file.evtx' WHERE Strings LIKE '%C:\\MyWebsiteRootFolder%' AND EventTypeName='Success Audit Event'" -i:EVT

All failed attempts to do something to a file/folder

“Who/what users have attempted to run an action against a file that they aren’t allowed to do” (maybe a hacker, testing your site)

logparser.exe "SELECT TimeGenerated, EventID, EventType, EventTypeName, EventCategory, EventCategoryName, SourceName, Strings, ComputerName, SID, Message FROM 'C:\LogFiles\example_windows_eventviewer_file.evtx' WHERE Strings LIKE '%C:\\MyWebsiteRootFolder%' AND EventTypeName='Failure Audit Event'" -i:EVT

Next up -> Scanning IIS Logs to investigate security incidents

While File system auditing is not something you want to turn on your entire servers disk drive, for instances where few writes occur (such as your website directory), setting it up and then using it to “play back the tape” on what has occurred on your server’s file system is a powerful tool to have in your arsenal.

What’s more important is that when you go to inspect these logs files, that you have an easy and fast way to query them.

Next we’ll cover how to do the same thing against IIS logs.

Up Log Creek Without a Paddle – Part 2: IIS Log File Investigations

8/23/2012 8:12:15 AM

When it comes to reviewing visitor site usage, server bandwidth usage, or forensic security investigations; IIS log files often hold the answers. Although as I'm sure you’re more than aware, gigantic text files can be hard to view let alone pull intelligence from. Investigating a website attack can be really daunting when looking at log files as an information source. In my previous post I covered a tool to help with Windows Security Logs. Lucky for us it’s just as awesome when dealing with huge IIS logs.

This is part 2 in a 2 part series on reviewing actual or investigating potential website security intrusions on IIS hosted websites. View part 1 here.

Image credit: Jim B L

Information overload…

IIS Log files are great as they store the timeline of all usage on your website. If you know where to look, you can retrieve information that helps draw conclusions on so many different visitor scenarios. In the context of security, if you’re armed with the knowledge of what to look for, IIS log files can be a great tool to forensically review an attackers behaviour; whether they have been passive in their prodding of your application by just sniffing around or active by actually trying to break things.

Fields of Gold

When setting up your website, IIS can log a significant set of data; but not every option is turned on. To turn on full logging, open IIS Manager, select your server’s root node and select “Logging”, then open up the “Select Fields” button.

Select every option to include as much data as possible.

Once you have this turned on, you’ll end up with a sea of information in these log files. On a busy website, these files will probably be too large to open in notepad.

Log Parser to the Rescue

As I covered in my previous post, opening and then scanning huge logs can be a daunting task – luckily Microsoft Log Parser 2.2 solves this problem with a tool that makes it easy to query multiple huge log files using well known SQL in record time!

Log Parser Lizard takes this one step further by allowing you to do this from a GUI interface.

To prove this point, load up Log Parser Lizard and write a simple query against the whole log folder (not just a since large file, but many) for one of your IIS sites:

SELECT TOP 100 * FROM 'C:\inetpub\logs\LogFiles\*WEBSITE LOG FOLDER*\*.log'

Pretty cool, eh?

So you’ve got log files, you’ve got a kick ass tool to query them, but what queries will help you make sense of the data or track down the evil doers?

First look: Security queries

To start with, you need to think of the context you’re looking at when querying your log files in. In issues where someone has attacked your site in the past, you’re probably in the search for answers on how.

If so, do you have an incident time to start trying to track down how it happened?

Getting a feel for when/how

Maybe try a query for a certain date/time:

SELECT * 
FROM 'C:\inetpub\logs\LogFiles\*WEBSITE LOG FOLDER*\*.log'
WHERE date = '2012-07-26' AND time BETWEEN TIMESTAMP('02:03:00','hh:mm:ss') AND timestamp('02:07:00','hh:mm:ss')

Or maybe you just want to start by looking for server errors as a start, to see if an attacker was testing your website for vulnerabilities?

Query looking for server Http response code 500:

SELECT * 
FROM 'C:\inetpub\logs\LogFiles\*WEBSITE LOG FOLDER*\*.log'
WHERE sc-status = 500

Another handy behaviour to look for is where you have a secure admin section. Most admin sections on websites redirect you if you aren’t logged in. They usually do this with a 302 redirect.

So let’s look for visitors trying to access your site using the relative URL “/admin/*” and who’re being redirected using 302 headers. While this will show some false positives for wherever you’ve tried to log in and forgotten the password, it’ll also give you an idea if someone is sniffing around.

SELECT * 
FROM 'C:\inetpub\logs\LogFiles\*WEBSITE LOG FOLDER*\*.log' 
WHERE  cs-uri-stem LIKE '/admin/%' AND sc-status = 302

Following a security trail

Once you have what looks like an intruder poking around, you then want to see what else they’ve been doing – and you should have their IP address from the previous query so we can query the logs for them:

SELECT *
FROM 'C:\inetpub\logs\LogFiles\*WEBSITE LOG FOLDER*\*.log'
WHERE s-ip='*IP ADDRESS OF ATTACKER*'

You can also see how much traffic is coming from single IP addresses by querying for each of their total usage – this might show someone attempting to brute force your site over time.

select TO_LOCALTIME(QUANTIZE(TO_TIMESTAMP(date, time), 3600)), count(*) as numberrequests 
from 'C:\inetpub\logs\LogFiles\*WEBSITE LOG FOLDER*\*.log'  
group by TO_LOCALTIME(QUANTIZE(TO_TIMESTAMP(date,time), 3600))
order by numberrequests desc

You can then query your logs for how many requests are coming from certain IP addresses to your “admin” section, again seeing brute forcing issues.

select TO_LOCALTIME(QUANTIZE(TO_TIMESTAMP(date, time), 3600)), count(*) as numberrequests 
from 'C:\inetpub\logs\LogFiles\*WEBSITE LOG FOLDER*\*.log'
where (cs-uri-stem like '/admin/%') 
group by TO_LOCALTIME(QUANTIZE(TO_TIMESTAMP(date,time), 3600))
order by numberrequests desc

Non security related queries

Checking for incoming broken referral links:

SELECT DISTINCT cs(Referer) as Referer, cs-uri-stem as Url
FROM 'C:\inetpub\logs\LogFiles\*WEBSITE LOG FOLDER*\*.log'
WHERE cs(Referer) IS NOT NULL AND sc-status = 404 AND (sc-substatus IS NULL OR sc-substatus=0)

Top 10 slowest pages/files to load:

SELECT TOP 10 cs-uri-stem, max(time-taken) as MaxTime, avg(time-taken) as AvgTime
FROM 'C:\inetpub\logs\LogFiles\*WEBSITE LOG FOLDER*\*.log'
GROUP BY cs-uri-stem
ORDER BY MaxTime DESC

Traffic by day:

Select To_String(To_timestamp(date, time), 'MM-dd') As Day, 
    Div(Sum(cs-bytes),1024) As Incoming(K), 
    Div(Sum(sc-bytes),1024) As Outgoing(K)
From 'C:\inetpub\logs\LogFiles\*WEBSITE LOG FOLDER\*.log'
Group By Day

Summary

Log Parser is an awesome tool for querying large log files. Log Parser Lizard makes this even easier with the addition of a GUI interface. For extremely large files I prefer to use the command line client for speed, but using the GUI to build your queries makes like just so easy – all of a sudden your information overload becomes a high signal/noise ratio sweet symphony that can help you either get a better feel for how your site is being used, or in the worst case scenario help you track down an evil doer and how they got in.

Visual Studio 2012 Web Deployment Projects are Dead – Long Live Publishing Profiles

8/26/2012 11:00:43 PM

I’ve been a long time supporter of Visual Studio Web Deployment projects. Not because I built ASP.Net websites and wanted to compile them, but more because they held so much unadulterated power from the simplicity of just being an MSBUILD file inside your solution. With the launch of Visual Studio 2012 Microsoft has made the call to no longer support WDP moving forward. This made me sad; but I was just being naive. Visual Studio 2012’s Publishing profiles are even more powerful, and they bring all your old friends along for the ride.

Blinded by the light…

A couple of weeks ago I attempted to rally some troops to fight the good fight.

“… Who took my cheese?…”
- WDP Lovers everywhere

I’ve been using Web Deployment projects to power the Continuous Integration of every project that my work has built in the last 3 years. I work in an agency where we release up to 10 sites a month. This is a lot of projects. And they all rely on Visual Studio Web Deployment projects to be deployed reliably from day one.

The removal of Web Deployment projects from Visual Studio 2012 worried me. Not “ get your pitch forks” level of worry, but more a “wow what a real shame”.

This is where you may say:

“…Hey guy, why use WDP when Web Deploy has been around for years. I thought WDP were just for ASP.Net “website” n00bs living in the past?”
- Web Deploy user

You may wonder why I love Web Deployment Projects so much? Well its quite simple really;

They sit nicely inside your Visual Studio IDE and were easily saved your deployment config to your source control.

Web Deployment projects are really just MSBUILD files, and therefore you can do anything inside them using the already outlined build tasks.

<Target Name="BeforeBuild">  
  <!-- // MAGIC UNICORNS ARE BORN HERE!! -->  
</Target>  
<Target Name="BeforeMerge">
  <!-- // MAGIC UNICORNS ARE BORN HERE!! -->
</Target>
<Target Name="AfterMerge">
  <!-- // MAGIC UNICORNS ARE BORN HERE!! -->
</Target>
<Target Name="AfterBuild">
  <!-- // MAGIC UNICORNS ARE BORN HERE!! -->
</Target>

As they’re just MSBUILD they are supported nearly everywhere by everything, and can use all MSBUILD extensions inside them.

<!-- 
Import the oober cool MSBUILD community tasks from
https://github.com/loresoft/msbuildtasks  
-->
<Import Project="$(MSBuildExtensionsPath)\MSBuildCommunityTasks\MSBuild.Community.Tasks.Targets" />

It’s also really easy to build different configurations for different environments simply through different Configuration Manager Solution profiles.

And you could script the entire deployment, including calling Web Deploy from inside the web deployment project, to avoid having configuration mixed into your build server’s config.

All of this is testable on the local dev machine.

But guess what?

I was worried about nothing. The new publishing profiles are really the exact same thing as Web Deployment projects, but actually more powerful.

It’s time to move on… Publishing Profiles are actually better

The new publishing profiles in Visual Studio allow us to do all of the above, without really giving up anything at all.

They site nicely inside the IDE and are easily saved to your source control:

Publishing profiles are really just MSBUILD files, and therefore you can do anything inside them you just by using the same build target names.

<TargetName="BeforeBuild">

</Target>
<TargetName="BeforeMerge">

</Target>
<TargetName="AfterMerge">

</Target>
<TargetName="AfterBuild">

</Target>

Because of the above, you can still user all your MSBUILD extensions.

<!-- 
Import the oober cool MSBUILD community tasks from
https://github.com/loresoft/msbuildtasks  
-->
<Import Project="$(MSBuildExtensionsPath)\MSBuildCommunityTasks\MSBuild.Community.Tasks.Targets" />

And its still just as easy to build different configurations using the IDE and test locally.

You can also call these publishing profiles from the command line!

C:\Code\MyWebsite>msbuild MySolution.sln /p:DeployOnBuild=true;PublishProfile=Deploy-Production;

Awesome things that get to join the party

As I mentioned above, publishing profiles actually add more power to this story, as they have a number of features that Web Deployment Projects missed.

They have native IDE support for building your publishing settings.

These get saved into your profile without touching anything you’ve put inside your profile such as custom build targets.

You can leverage all of MS Deploy to plug in at any step of the deployment.

This allow you to do things like customise app_offline.htm files during deployment, and anything else you have in mind. Package a deployment package and check it into your source control? easy.

You can also run tiered web.config transforms

This allows you to run a transform for “debug” builds that maybe does things like turning custom errors pages off and then another transform for your “Staging” environment that sets your database connection strings up correctly. This is powerful. I used to have to do this level of granularity manually in an “AfterBuild” target in my web deployment projects. Now it’s there out of the box.

The future looks bright

If you were like me and were worried about Sayed’s post it’s time to calm down. There is no fire – in fact publishing profiles actually are more integrated, more powerful, and are going to be more widely used than web deployment projects ever were. This will mean great things for developers wanting to get into using Continuous Integration with ASP.Net as there will be lots of documentation floating around from different people’s take on what works for them.

If Continuous Integration and Deployment are your pot of gold, Publishing Profiles just might be the rainbow that’ll help lead you there. Well done Visual Studio team.

The Senior Position Fallacy

9/6/2012 8:25:18 AM

In the IT industry employees experience a weird phenomenon once they begin to move up the ranks. You often start work in IT because you get to build stuff, monitor things, and watch your creations grow. The weirdest thing about this is that in our industry to step up in your career you often have to actually stop producing things. To move into management put down the tools, and loosen your grip on what you love.

I’ve been working in IT my entire career. I started building websites and helping fix networks while I was still in high school. My first well paying job was actually working for myself growing a little digital production business. I learnt many things along the way, but I was lucky to realise quite early on that I would much rather be paying someone else to do the managing and accounting; leaving me to stay close to what I loved. I wasn't yet ready to move up the ranks.

Fast forward to today and I am now a year and a bit into my current role as a Technical Director. I take care of a development department.

When I was first offered the role it was an exciting opportunity that I craved. It was the chance to prove to myself what I was really able to accomplish if I had the reigns. I had worked in positions in smaller workplaces that carried similar responsibilities, but they were never officially recognized as such; being the guy in charge. It was an opportunity for me to take everything I had seen go wrong in a development department, and try and correct it little by little.

I was very aware while taking the position that in order to be successful in my new role I would have to give up a huge part of what made working in software my career for so long so that I could concentrate on my new place in the world entirely.

I had to step away from day-to-day development.

Step away from the tools. Close my IDE, and watch from afar (albeit only a few metres). I can’t lie and say this wasn’t hard. I have thought about it most days. At work if I'm talking development or tooling I’m talking architecture and application design. I still write a lot of code, I just often write most of it at home late at night – I’m “keeping the dream alive” so to speak. I’d never stop doing it completely, as apart from keeping current being important to my role, I think I'd actually go mad if I didn’t. I must state that I do really enjoy my new role though; it both challenging and rewarding. I still solve problems, just different types.

The level of recognition I have received has definitely risen though.

After promotion to my current position my life changed considerably. I won awards both for my department’s work and also personally from magazines and industry bodies for being “a guy on the rise”. I am asked to give my opinion on matters and more recently speak at a number of industry events. I’ve also been lucky enough to garner a higher salary. In some way, it’s almost as if I've been rewarded for stepping away.

Our industry isn’t the only one in which this situation occurs. You see it in many industries where people produce something.

And the whole experience has left me with a lasting question;

“Why does the world often only start to measure success, the minute you begin to step away?”

I understand there is a million reasons as to why this occurs, and I've definitely over simplified the issue; in some ways the question seems silly, but it still runs through my mind a lot.

Senior technical roles actually consist largely of responsibilities that are non-technical.

Microsoft TechEd Australia 2012

9/10/2012 11:55:47 PM

Today marks the official start to Microsoft’s TechEd Australia Conference on the sunny Queensland Gold Coast. With over 4 days of talks, product launch education, hands-on labs along with device and software manufacturers spruiking their wares, it is sure to be a great week – if you are around shoot me a tweet so that we can try and cross paths during the week.

I did say today is the official launch as today is the day the majority of delegates arrive - nearly 200 sleepy souls started their conference early yesterday by making it through the 24 hour straight Windows 8 AppFest for the chance to win a $1000 Windows 8 device. Finishing up at 11am this morning most are probably back at their hotels catching up on some much deserved sleep.

Today is a big year for Microsoft as it releases a whole bunch of updates to products that many developers and IT admins alike spend most of their days with. Their will be the new server line releases with Windows Server 2012, a big push towards Microsoft’s Cloud services platform Azure, much talk of their new IDE Visual Studio 2012, as well as a ramp up towards the release of Windows Phone 8 plus much, much more. There are so many different things going on that the program for the week feels more like a music festival guide than a IT industry conference timesheet, with thoughts running through my head like “Which stage are we going to next?” and “Why did they have to book these two things at the same time…!”.

Swag time

When the tickets to Microsoft’s Build 2012 event sold out in 2 hours, many said it was the chance that they’d be handing out Microsoft Surfaces as part of the ticket. TechEd has a history of handing out the obligatory Tech Ed Bag and a few associated deals packed in, they have sometimes also included devices along the way (such as the HP Mininote in 2009). This year the inclusion of a pretty epic Targus bag is no exception. This bag rivals “The bag of holding” with the amount of pockets included (I counted 12 zip pockets. Someone even thought they could get an extra one on the product sheet by including a zip pocket on the bottom of the bag?).

The inclusion of a water bottle to hydrate after any 24 hackathons along with a padded laptop section for yours truly to cart around my ThinkPad are definitely appreciated.

The TechEd 2012 “Swag bag”

On with the show

All in all I'm pretty excited about what is to come. I’m also looking forward to touching base with a number of others in the tech community and sharing ideas – if you follow me on twitter, or subscribe to my blog, shoot me a tweet or email and come say hello.

Setting up a VPN server on Windows 7 or Windows 8 – Secure your Internet use while away

9/11/2012 9:52:04 AM

These days we’re lucky. SSL is becoming pretty pervasive. Facebook uses it. Twitter uses it. Most modern start ups now use it. Sadly there are still other sites or services that you may be accessing on the internet that are still insecure allowing others to listen in on your internet usage, and for these your want an encrypted VPN link to route your traffic through. VPN’s can be expensive though if all you have is a home PC and a laptop on the road – lucky for us this can be a magic combination that is all you need and saves the day.

This week I’m in TechEd Australia. Microsoft make the fact that using the shared Wi-Fi could be fraught with peril:

The Australian TechEd program guide on Wi-fi usage.

The above statement didn’t worry me though. Why? Because I have a VPN to connect to when using third party internetzes while I'm on the road.

A few people I spoke to on the day seemed to think this was a lot harder than it is to setup. You don’t need to be a network engineer, and you don’t need Windows Server or a remote Amazon instance, or really much of a clue. All you need is a Windows 7 or Windows 8 PC to host your VPN (i.e. at PC at home), a Windows PC to dial in from (from XP right through to Win 8) and port forwarding support in your router.

How is this possible?

From Windows 7 onwards there is native support for hosting an incoming PPTP VPN service.

This allows you to setup a remote Windows 7 or Windows 8 machine as a VPN server for you dial in to while you are on the road, so that you can route your internet traffic through this remote machine’s internet connection in an encrypted fashion.

This allows you to cut the complicated setup of a whole bunch of services such as DHCP, VPN, and routing into a simple step by step walk through that can take you 5 minutes.

What you’ll need

Windows 7 or Windows 8 remote machine for hosting the VPN connection.
The ability to route internet ports directly to this machine (i.e. port forwarding support in your router or a PC connected directly to the internet).

On with it then…

At home I have a Windows 7 PC used as a media pc that is on all the time for media sharing and TV watching duties. This is perfect for me as it gives me a remote PC that is on all the time and is connected to an internet connection I trust. If you have such a PC and control the port forwarding to said machine, this is all you need.

Step 1: setup the VPN server (on your host machine).

The following works on both a Windows 8 and a Windows 7 machine (I've tested both successfully and the interface is exactly the same with just different “chrome” on the windows).

Open Network Connections

Hit the ALT Key to show the file menu, and then select New Incoming Connection.

At this point you can either select one of the current local machine users and grant them access to your new VPN link, or take my suggestion and create a new user just for VPN access and give it a really strong password. This will ensure that even if an evil doer gets into your VPN link, they don't necessarily have any of your other account files with the same account. Don’t make it any easier for them.

Then on the next page, tick the box that mentions that users will be connecting “through the internet”.

On the next page tick the network protocols you would like them to have access to. I’ve left mine “as is” as I only need IPv4.

Then click “Allow access”.

This has finished the setup of your server.

Step 2: Setup port forwarding.

Next you need to allow “the internet” to talk to your host PC on TCP port 1723.

First setup a static IP address for your host PC on your local network.

Take a look at Port Forward to find your router and instructions for how to forward TCP port 1723 to your machine’s IP address from the internet.

This will allow your host PC to be contactable from the internet, but unless you are lucky enough to have a static internet IP address from your ISP, or you don’t mind having to remember IP addresses, you’ll want to give your host machine a nicer address.

To help with making your host PC easier to connect to, take a look at setting up DynDNS.

Step 3: Your client PC.

Now that you have your home host PC all setup, you simply need to setup your client PC to connect.

This is just as simple.

Open the network control panel item Setup a VPN Connection.

Enter the remote hostname for your PC. If you’ve setup DynDNS this is your DynDNS host address (ie [yourhostname].dyndns.org.

Click Create.

Now when you attempt to connection, simply enter the username and password you created earlier (the one with the really strong password) and connect.

Now enjoy the security of having a remote VPN setup without all the server management hassle.

You have been warned

Using 3rd party Wi-Fi at your local cafe or in my case conference centre is a bad idea. Any website you visit while connected to these Wi-Fi access points risks anyone else on the network sniffing your traffic and stealing your session using something like Firesheep.

However there is another risk in setting up a VPN by following the above:

By following the above you are directly placing your PC “on the internet”. This means the evil doers can have just as much access as you. Be sure to place a really strong password on any accounts you setup, and change it regularly. You can also look at using a different TCP port for your VPN so that there is a bit more obscurity for anyone just trying to brute force port 1723.

Problems with .Net 2.0 Mixed Mode Assemblies inside Visual Studio .Net 4.5 Test Projects

9/13/2012 12:22:50 AM

When you include .Net 2.0 mixed mode assemblies in .Net 4.0 or .Net 4.5 projects you often have to add some start up options to your project’s config file to get it all to play nice when your app starts up. This backwards compatibility feature is great as it allow you to use older/non supported projects in your recent work.When when using Visual Studio 2012’s new unit testing tools this magical piece of app.config code doesn’t seem to help though, and the solution is pretty simple.

Recently I’ve had great fun working on some green field .Net 4.5 projects. While Unit Testing some of the code in these projects using Visual Studio’s “super sweet” async testing tools I’ve come across a niggling little issue that breaks with normal convention when it comes to testing code that depends on older assemblies.

The Problem

Normally when you are loading .Net 2.0 assemblies as a reference into a .Net 4.0 or 4.5 project you have to tell .Net that it should run in a compatibility mode so that your old assembly will load properly.

This is because .Net 2, 3.0 and 3.5 assemblies are all built against the .net 2.0 CLR.

.Net 4.0 and 4.5 assemblies are built against different CLR’s and therefore startup differently.

For backwards compatibility the Microsoft .Net teams kindly allowed you the add the following to your web.config or app.config to make this all still work:

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
  <startup useLegacyV2RuntimeActivationPolicy="true">
    <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.5" />
    <supportedRuntime version="v2.0.50727"></supportedRuntime>
  </startup>
</configuration>

The above will make your application run fine when placed into the app’s config file.

When writing unit tests against a .Net 4.0 or 4.5 assembly and running these unit tests inside Visual Studio 2012 this fix sadly doesn’t work.

Case in point with my SharpSVN “bang my head against a brick wall” exception below which worked when running in my app, but failed under unit test (the be fair this is an integration test – apologies to the unit testing elitists out there…):

The Solution

The reason the normal app.config fix doesn’t work when using the Microsoft Test runner built into Visual Studio 2012 is that the test runner’s environment isn’t inside your test project’s bin folder. It’s running somewhere else.

The file you need to place your compatibility config settings in is:

C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.executionengine86.exe.config

If you open the above file using your favourite notepad (running as Administrator) and place the same “startup” section mentioned higher in this post inside this file, all your problems resolve themselves.

Hopefully the above post can save you the time it took me to find it.

Integration Testing FTP Connections in .Net

9/26/2012 7:16:32 AM

When writing testable code your first port of call is often to abstract any dependencies and make them easy to mock. This is the same for any of your codebase that talks to FTP servers. Testing the way your code behaves under real world conditions makes integration tests important regardless of abstraction though. Here’s a simple trick to test FTP code in the wild.

A recent project of mine has involved writing code that talks to FTP servers with the goal of adding additional continuous integration automation to a project. Although all of my main methods are easily abstracted and injectable, my project still needs to actually talk to FTP servers at the end of the day, and I need to test that these very methods do the right thing when they are met with different conditions; be they bad credentials, lack of read/write permissions etc.

The Challenge

Integration tests can be brittle at the best of times, so ensuring that they are repeatable and can be setup and torn down can often be almost as much of a challenge as writing your actual code itself.

An FTP server is usually a static service that is installed on a server. You might think that running one and ensuring it stays up and doesn’t get hacked just so that all your integration tests work is a necessary evil, but there is an easier way.

Run local. Run often.

I was running an FTP server on my build server just so that it was “always around” for my tests until i stumbled across an interesting project over on GitHub to do just this.

The approach I'm about to show you doesn’t need you to go to the effort of running a dedicated server at all. All you need to do is add a single executable to your unit test project and wrap your unit test in a using statement.

The FTP server executable is a single file FTP server called FTPDMIN which offers a read/write FTP server that can be fired up from the command line with a minimum feature set and only a few command line parameters to make it all tick.

By implementing IDisposable the helper class that wraps around this command line exe allows you to take advantage of the using() pattern to take care of your executable’s lifetime and have it die when your code is done testing.

Steps to make it happen

Download FTPDMIN from here.

Add the exe to the root of your test project (you can put this anywhere, but you’ll have to update the helper class below).

Now add the exe to your project (i.e “view all items” in your test project’s solution explorer, and add the exe).

Set the EXE to “Copy always” in it’s solution properties.

Add the following code to a helper class in your Test Project:

public class FtpTestServer: IDisposable
{
    private readonly Process ftpProcess;

    public FtpTestServer(string rootDirectory, int port = 21, bool allowUploads = true)
    {
        var psInfo = new ProcessStartInfo
            {
                FileName = AppDomain.CurrentDomain.BaseDirectory + "\\ftpdmin.exe",
                Arguments = String.Format("-p {0} -ha 127.0.0.1 \"{1}\" {2}", port, rootDirectory, allowUploads ? string.Empty : "-g"),
                WindowStyle = ProcessWindowStyle.Hidden
            };
        ftpProcess = Process.Start(psInfo);
    }

    public void Dispose()
    {
        if (ftpProcess.HasExited) return;
        ftpProcess.Kill();
        ftpProcess.WaitForExit();
    }
}

Now you can enjoy being able to write really clean integration testing code that starts and FTP server every time you run your tests and then tear it down when your test is done.

An example integration test showing connecting to “127.0.0.1”:

[TestMethod]
public void FtpCode_Upload_CanConnect()
{
    try
    {
        // Fire up a new Ftp server instance
        using (new FtpTestServer(rootDirectory: "./"))
        {
            // code that talks to an FTP server on 127.0.0.1
        }
    }
    catch (WebException e)
    {
        Assert.Fail("Failed to connect to our FTP server");
    }
}

How awesome is that?

The power of using FTPDMIN is that it can be told to deny write permissions to simulate bad user permissions as well:

[TestMethod]
public void FtpCode_Upload_ThrowsWebException()
{
    try
    {
        // Fire up a new Ftp server instance
        using (new FtpTestServer(rootDirectory: "./", allowUploads: false))
        {
            // code that talks to an FTP server on 127.0.0.1
        }
    }
    catch (WebException e)
    {
        Assert.Fail("Our code failed to upload a file because of invalid permissions");
    }
}

All in all the above has been a complete life saver when it comes to making my integration test projects portable – if a new developer joins my project, they instantly get access to my FTP test harness just by pulling down my project’s source code.

The Pitfalls of Cut and Paste Coding

10/2/2012 7:51:49 AM

We’ve all been guilty of it in our development careers at one time or another. When starting out using a language or framework that you’ve never used before you often have no choice but to. What I’m talking about is the act of “copy paste coding”, and it’s as common in the programming world as chewing gum under seats. When you copy and paste other developer’s code into your application it’s important to fully understand what the code does before you continue; or risk joining the many fools that have gone before you.

When I was in primary school there was a really smart kid in our class; let’s call him Joshua. He got top marks pretty much all the time. My class also had a not-so-smart kid called David; to be fair there were a few Davids in my year group in primary school. David was the kind of kid who’d do anything for a shortcut, so he used to sit horizontally behind Joshua and take notes from his test sheets every time we had a Maths test. Primary school obviously seemed a lot harder when you were 10.

This went on for a couple of months; remember that I mentioned that Joshua was a pretty switched on kid and hadn’t missed the gist of what was going on. So one day we had another test and we all sat down and got stuff in. Joshua smashed the test in only 10 of the 60 minutes we had been given – to say he made us all feel inferior when he put his pencil down was an understatement.

After a slight pause, he then preceded to pull out a second test sheet and really start his test. Joshua had taken two test sheets and written fake answers to all the questions in the test.

The look of shock on David’s face was priceless.

Time to go on Safari

Recently a lot of the sites that I look after have been receiving some serious ELMAH spam. So much so that I've had to implement ELMAH filters just to get the Signal-to-noise ratio of my application’s exceptions back to a manageable threshold.

The offending exception that I've continually seen:

System.Web.HttpException: A potentially dangerous Request.Path value was detected from the client (:).

So someone is making bad requests to my web applications.

From looking at my ELMAH logs over time this is from a number of different sources ion different IP address ranges around the globe.

All of them have the same pattern to their exception.

Request Path Info: ”https:/mywebsite.com/”
User agent: ”Test Certificate Info”

They are all missing the second “/” in the https:// of my SSL site’s addresses.

So I think to myself:

“Why the hell does every site that I manage that uses SSL seem to be getting the same exceptions?”

My exceptions where all for different websites on a number of different servers.

So someone out there is scanning websites with a piece of code that puts bad paths into URLs, and uses the user-agent string “Test Certificate Info”.

Following the paper trail

So I set out on the interwebs in search of answers.

I can’t be the only one experiencing this?

And It turns out that I’m not. It seems that more than a few of us are experiencing the same thing.

One of the responses to the above linked StackOverflow post seems to nail what has happened:

“…
My guess someone read this and didn't end up changing the example code.
…”

The link in the quote above is to an MSDN C++ article that walks through making an SSL request – but until recently the code sample on the MSDN post had an error that wrote an invalid Http header while making its request.

And therein lies the problem.

A copy of a copy, of a copy, of a copy

Just like my not-so-intelligent primary schoolmate David mentioned earlier, when whoever wrote the piece of software that’s been visiting my sites was cranking out code in a haze of Doritos and Pepsi, they copied someone else’s work – and with it they copied a bug.

Unlike David however when you’re a developer this is pretty normal behaviour and wouldn’t be considered unsavoury at all – hell, open source breeds on the act of copying.

However there is one thing that like David we all have to be careful of, as copying other developers code from the internet does have it’s own set risks that you need to be aware of.

If you copy someone else’s code and you don’t fully understand what it is doing, then you probably aren’t in a place to use it in a production environment.

Whoever wrote the web spider that is out there indexing all my SSL website’s and causing my ELMAH error spam obviously didn’t note the issue in the code sample they copied from. Like them if you do the same, you might end up with a piece of code in your application that simply makes you look like an amateur.

And it doesn’t have to be that way.

Jeff Atwood and John Galloway started adding the following badge of certification to code samples nearly 5 years ago:

The above image was basically a sarcastic joke at the fact that a lot of code in world has simply not really been properly tested. And I don’t meant test-driven, test-first level tested – I mean the level of testing where it hasn’t really ever been used outside of simply knocking together a proof of concept sample for a blog post.

I’ve been guilty of this before when I’ve posted things on this very blog that have needed an update after the fact.

So if you’re out there copying code from other developers, open source projects or blog posts – make sure you read the logic you borrow thoroughly, take the time to understand it, and for your own sake make sure you test anything that spiders websites all over the internet before you hit the launch button.

My event logs will thank you.

Teaching Your Application to Read

11/28/2012 4:03:06 AM

Geeky salesman have spent decades selling us a pipedream of the end of “Tree based communication”. To date not much has changed and I’m not holding my breath. In the mean time, what do we do with all this paper? If like me you are dabbling with ways to get your code to convert the physical to the digital, I’ve got news for you: Smarter people have already solved the problem for you and it’s simply a matter of plug and play.

There is part of the world that us computer nerds often tend to hate. Hard copy. A physical information format you can actually hold in your non-Neo physical hands, Paper. I haven’t just gone out and become a print/paper guy overnight, but I have been playing with a project that takes paper in and gives digital goodness out, because whether you subscribe to reality or not, the plain and simple fact is this medium isn’t going anywhere soon.

Let me get my glasses

Building apps that behave like humans is hard. Building ones that see the world and interpret it like we do is even harder – it’s because of this that developers at Google, Microsoft and many other large companies have invested so heavily in projects like reCAPTCHA so that they can try and teach computers to read the way humans do.

But what if you happen to not have millions in research funding, computer scientists filling your development teams, and still want to add the ability for your app to read documents and structured handwriting?

This is where OCR software comes in, and more directly for us developers, OCR software that offers an easy to use API for plugging your app into its 20/20 vision. A great tool for achieving this is the ABBYY FlexiCapture SDK which allows you to quickly add both document OCR, and even more interestingly document recognition and categorization to your apps. This means not only does your app quickly gain the ability to read documents, invoices and printed handwriting, but also the ability to act like a mail clerk and categorize any of your documents for later storage and review.

Standing on the Shoulders of Giants

Whenever you plug in a third party API into your project you get a pretty good view of how much they’ve thought about how us lowly code monkeys will consume their product/service. To put a point on it, less is more, I want to get up and using a product with as little friction as possible. The chaps at ABBYY have ticked the box here and you can get a feel for their API by taking a look at what is involved in converting the following image into machine readable XML. All I have to do is scan the image once to create a Document Definition with parts of the document I’d like to capture and its off to the races.

[DllImport("FCEngine.dll", CharSet = CharSet.Unicode), PreserveSig]
private static extern int InitializeEngine(string devSN, string reserved1, string reserved2, out IEngine engine);
[DllImport("FCEngine.dll", CharSet = CharSet.Unicode), PreserveSig]
private static extern int DeinitializeEngine();

void ProcessImage()
{
    IEngine engine = null;
    InitializeEngine("SWAT1000000000000000", null, null, out engine);

    var processor = engine.CreateFlexiCaptureProcessor();



    //Add our definition file
    processor.AddDocumentDefinitionFile("LicenceseDefinition.fcdot");

    //Add an image to scan
    processor.AddImageFile("license.jpg");
    var document = processor.RecognizeNextDocument();

    //output a file "LicenseOutput.xml" with the results of the scan
    processor.ExportDocumentEx(document, "MyOutputFolder", "LicenseOutput.xml", null);
    DeinitializeEngine();
}

Output:

<?xml version="1.0" encoding="UTF-8"?>
<form:Documents xmlns:form="http://www.abbyy.com/FlexiCapture/Schemas/Export/FormData.xsd"
                xmlns:addData="http://www.abbyy.com/FlexiCapture/Schemas/Export/AdditionalFormData.xsd">
    <_License:_License xmlns:_License="http://www.abbyy.com/FlexiCapture/Schemas/Export/License.xsd">
        <_License>
            <_Name>McLOVIN</_Name>
            <_DriversLicenseNumber>01-47-87441</_DriversLicenseNumber>
            <_AddressLine1>892 MOMONAST</_AddressLine1>
            <_AddressLine2>HONOLULU, HI 96820</_AddressLine2>
            <_IssueDate>06/18/1998</_IssueDate>
        </_License>
    </_License:_License>
</form:Documents>

As a Web Developer by trade I don’t spend much of my time close to the bare metal, so although the use of calls to unmanaged code shown above makes me feel a little dirty, the fact that I only need such a tiny amount of code to get me to a working solution makes it even more worth my while.

And before you go thinking I’ve spent hours and hours with the product, luckily for me ABBYY has been nice enough to provide a huge bunch of working code samples in their SDK pack. Code Samples are the commercial equivalent of providing access to their unit tests so you can see how to actually build stuff. Less than 10 minutes in, and I have my app scanning fake Hawaiian drivers licenses – Awesome!

Although I won’t be covering it in this blog post, one of the more exciting features that comes along with the FlexiCapture SDK, is the ability to self learn documents that you feed it. As if all those warnings Sarah Connor gave us had fallen on deaf ears…

Input/Output

You’ll note that in my code above I load a JPEG image into the FlexiCapture SDK as a source, but if you’re scanning documents, screenshots, video frames, or even a blurry smart phone photo the FlexiCapture SDK can take in a feast of formats in all sorts of quality and DPI:

PDF
Bitmap
PCX
JPEG Black and white
JPEG 2000 (12 subformats)
TIFF (10 subformats)
GIF (Because we all take high rez photos/scans as GIF these days)
PNG
DjVu
JBIG2

This means you can easily write applications that take all sorts of image or scanner data in and give you easy to read data out – you can easily see how receiving faxed, emailed or uploaded images from disparate sources becomes a lot easier for your application to make sense of. Once its taken a peak at your incoming data sources it can then pump data out in a whole bunch of other formats that support structure data. Take a look at my McLovin output in a bunch of formats:

CSV Text File

Excel File

A full dBase Database (thanks to DBF Viewer Plus for allowing me to even see this!)

It also allows export to XML (shown in my initial classy sample), and a whole heap of image formats as well – although seeing you’ve just imported an image this might not come in overly handy unless for archival.

You do get the picture though – ABBYY FlexiCapture isn’t just a one trick pony, but more a Swiss Army knife for character recognition and image to text scanning.

What makes this even more powerful is that it does all of the above recognition input/output trickery in 198 languages.

That’s right, I said:

198 languages!!!!

This is priceless when you consider that 91% of the world doesn’t speak (and therefore write) in English so I'm blown away by this number.

Included Tooling

We really should have listened to Sarah Connor, as one of the coolest features of the ABBYY FlexiCapture SDK is that it auto learns documents for later classification. If you are scanning large amounts of documents, it can learn to read and classify new types of documents simply by you feeding them through the API – handy for times when a supplier or client changes the design of their invoicing.

For everything else there is a great set of included tooling, with apps for classifying documents, defining new document definitions, and as a reviewer making changes to captured text that might not have the best confidence level; They even have a form designer in case you would like to create new forms for later scanning.

The review console

The document definition studio for defining fields

Form designer in action

Summary

If you’re working on a .Net project that needs to consume paper data sources such as forms, invoices, or emailed ID information for later review, the ABBYY FlexiCapture SDK is definitely worth taking a look at for your next OCR project. I know that the speed to get up and running really did make my life easier, and all of the tooling that comes along with it made the product feel like a very complete solution. As a c# guy the only negative I would say about the whole experience is the fact that the c# API requires the use of non-native code, although when considering the amount of image processing and heavy grunt work that the API must be doing in the background to pull off it’s magic I understand why this decision was made.

Now we just need to figure out a way to stop Judgement Day.

I received the product mentioned above for free in the hope that I would mention it on my blog. Regardless, I only recommend products or services I use personally and believe my readers will enjoy. I am disclosing this in accordance with the Federal Trade Commission's 16 CFR, Part 255: "Guides Concerning the Use of Endorsements and Testimonials in Advertising."

Reaching The Internet Event Horizon

11/29/2012 2:57:16 AM

For the last 15 odd years we have continually expanded the amount of information available at our finger tips year on year exponentially. Thank you Internet for all your glorious contributions to our lives. Only recently has staying connected to the internet on a Mobile really started to take this to a level where I can truly say that I really am "connected on the go" and as far as technological achievements go that is a pretty amazing achievement to witness in our lifetimes.

When I first connected to the internet I did so by walking down to my local computer store down the street and purchasing prepaid vouchers from a local ISP called Microplex. It was one of those small hole in the wall white box stores of the late 90’s. As a teenager obsessed with computing this local PC store was my own little technological Mecca. For the value price of $39.95 my pocket money unlocked 10 hours of unlimited #EFNET IRC chats and late night school assignment “assistance”.

All at the break neck speed of 28.8kbp/s.

I needed a helmet for safety.

The world didn’t stand still though, and I definitely wasn’t going to intervene, so incrementally over the next very short few years I found myself taking up the past time of buying modems like they were fashion accessories.

33kbp/s.

56kbp/s.

56kbps/s V.90.

K56kbp/s.

128kbp/s ISDN.

1.5mbp/s ADSL.

5mb/s Cable.

24mb/s ADSL 2+.

It was an ever exciting race to take part in. I’d research ways to test and ensure my phone line had the least amount of noise possible, and learnt all about things like central filters, and ISP’s had dedicated lines for different types of modems to help you squeeze as much juice as possible out of your 56kbp/s. The most interesting thing I noticed in hindsight was that somewhere between the excitement of Cable internet and ADSL 2+ something funny happened. The Internet stopped mattering. In the is-it-a-busy-time-of-day, am-I-waiting-for-a-page-to-load, Im-so-angry-someone-else-got-that-e-bay-item, always connected frame of mind it simply faded into the background. The Internet was so fast that you stopped even thinking about connectivity, let alone whether anyone else in your house was about to pick up the phone.

“…

Mum!

Get off the phone, I’m in the middle of a game of MechWarrior against some British n00bs!

…”

I didn’t take any notice at the time but a glorious thing had occurred.

I had won the internet.

Or so I thought.

Welcome to the Future

In a similar but more recent time frame another speed race has also been taking place in pockets and hand bags everyone. The race for Mobile Internet speed.

In the past 5 years or so, Internet access on the go has moved from being a useless add-on to sell you expensive Telco plans, to being a serious value add for anyone. Staying obsessed with your friends, family, and sad-to-say work email around the clock would never be possible without Mobile Internet.

As Scott Hanselman so eloquently put it we’ve become infatuated with the idea of it:

“…

Just pull to refresh

…”

It has become so popular to surf the Internet on your Mobile that society is becoming all too familiar with any number of new social faux pas that surround smart phone usage. Companies are even doing interesting studies to uncover meaningful nuggets like the fact that 19% of people drop their phone down the toilet.

But I’ve still found Mobile Internet awareness to consistently be front of mind for me. Having built a number of data centric mobile apps over the last few years I’ve found that whether building apps or consuming them, the mobile internet offered by GPRS, 3G, and HSDPA+ has always made me critically aware that I was “using the Internet on my mobile”. My brain was always coming up with more questions than mobile Google could begin to answer fast enough for me on my morning commute.

Until today.

I give you: 4G LTE.

After firing up my new Nokia Lumia 920 for the first time the other day I experienced something I’ve never felt since using the Internet on a mobile, I stopped being even aware of the internet. I’ve spent the last 3 days downloading, streaming, Google’ing, Skyping and generally just getting stuff done on the internet. And it’s all been flawless. Obviously being on my Island nation’s fastest Telco and sticking to built up areas has helped a lot with this (thanks 90’s anti-competitive Telstra monopolism), but I simply couldn’t care less. I now have faster internet on the go than I do at home and living across the road from the local phone exchange has made my ADSL 2+ connection nothing to snicker at.

My new Lumia 920 now downloads at an average of about 39-40mb/s (from Doug’s oober thorough testings).

That’s 1,388 x faster than when I first started using the Internet.

This is not only summarised as fast when compared to my home internet connection, but in human terms this is revolutionary. It means my main daily input source for information is actually able to give it to my faster than my poor little evolved brain can consume it. I am able to trigger a human IOPS issue anytime day or night on command.

I’m pretty sure Science would define this as meaning I have reached the Internet Event Horizon.

And as Sam Neill taught us all, when this happens there is simply no turning back.

Entity Framework Code First Migrations in Teams

12/4/2012 6:09:08 AM

Migrating to a Code-First or Model-First approach to database development can be very liberating. At the end of the day your database is just a way of storing state, so getting away from the implementation details can really help speed up development and allow you to focus your efforts. Code First’s awesomeness aside, when you try and implement this kind of paradigm shift within a team you unlock a different set of problems. Here are two potential ways to alleviate some of the headaches.

Entity Framework has had a Code First approach for a little over 2 years. In that time if you’ve been surfing the net and read all the posts and reviews covering peoples experiences, you’ll come away with the view that it has become quite popular with developers everywhere. One thing you don’t see many people talking about however is how it works to use it in teams of more than one. It isn’t suited to all projects that is clear, but when used correctly it can really speed up development.

Implementation Details – Removed or Replaced?

Entity Framework Code First Migrations force you to think about your database model differently. Instead of thinking tables and columns, you simply think about your domain objects as they are - the rest will “sort itself out".

The “old school” database first approach

namespace Blog.Data
{
    public class BlogContext : DbContext
    {
        public ICollection<Author> Authors { get; set; }
        public ICollection<Blog> Blogs { get; set; }
        public ICollection<Category> Categories { get; set; }
    }
    public class Author
    {
        public string Name { get; set; }
        public virtual ICollection<Blog> Blogs { get; set; }
    }
    public class Blog
    {
        public string Title { get; set; }
        public string Content { get; set; }
        public string Url { get; set; }
        public virtual Author Author { get; set; }
        public virtual Category Category{ get; set; }
    }
    public class Category
    {
        public string Name { get; set; }
    }
}

The “new” sexiness/“Domain model is all I care for” approach

As you can see from the second approach allows you to simply stop caring for how the data is even stored, and just concentrate on your app and the model of your data as classes and properties. You create your domain objects and Entity Framework takes care of the rest. Like a lot of the additions to .Net over the past few years, this is not new, but rather borrowed from other frameworks like Ruby On Rails.

What it’s likes when working alone

When working with Code First Migrations, your work flow reeks of awesome simplicity:

This is a dream to work with, and when you go to push between staging and production the roll up is all taken care of for you, and you simply get on with better things. This is the magic of Code First Migrations!

The problems only begin to appear when you have more than one developer working with in a team. This is even more of a problem if your team aren’t within close enough proximity or contact to talk all the time as they can’t communicate of segregate who is updating the domain model and migrations at a time. In fact if you are using anything more than a rudimentary approach to source control this is next to impossible as you’ll always be merging codebases that started in a shared previous state.

This then looks more like:

Did I mention that Code First Migrations does some magic behind the scenes that makes each migration aware of its “previous state” making it next to impossible to “merge” in the way you would have previously?

When these migrations are then played out against your database, developer B (or whoever checks in last) and their migrations always fail as his/her migration comes from a parent domain model state that no longer exists, as the current version of the model has changed since checkout or branch.

Unlike Ruby on Rails migrations, the code migrations generated by the PowerShell helper in EF Code First add an additional field that make them database state aware.

We’ve replaced one set of implementation details with another.

Magical voodoo in each of the migrations:

string IMigrationMetadata.Target
{
    get { return "XiSR4t76Fv...G2vKGQAAA=="; }
}

This field contains an encoded string representing the EDMX XML model of the domain model at time of migration. Entity Framework checks your database before applying every migration to ensure that the database is in the state that it thinks it was from the previous migration classes Target property. This then stops migrations from occurring on a database that has changed since checkout, to avoid any data loss or confusion over elements that may have changed in both migrations. It then stores a copy of this serialised model in the system table __MigrationHistory.

Code First Migrations in a Team

The following two methods are process solutions to this problem. They aren’t perfect, but they do seem to make the workflow a little less frustrating at times if everyone in your team is aware of the steps and follows them – very similar to some of the workflows for branching and rebasing used in Git or Mercurial these days, after a while these steps can second nature.

Both methods rely on local development databases being used by each developer separately. This approach therefore also requires seeding any testing data or 3rd party data.

Action Plan A - Automatic Migrations

The first method is based around the use of Automatic migrations while working on your local changes, and then pulling your repo down, merging any domain model changes and creating a new manual migration.

This method works like:

Pull down the latest Domain Model into a branch.
Turn on Automatic Migrations.
Develop, Develop, Develop.
Blow away your Development Database (and with it your __MigrationHistory table, and it’s state)
Turn automatic migrations off
Pull the latest trunk/master Domain Model from your Source Control repo up into your branch.
Merge the latest Domain Model with your local changes.
Add a new migration of “the difference” between the repo and your domain model.
Test a migration “up” on your local DB.
Merge your branch back into your mainline.

Action Plan B - Manual Migrations

This approach is quite similar but works to allow local code first migrations during any given modification period.

This method looks like:

Pull down the latest Domain Model into a branch.
Develop, Develop, Develop - adding as many migrations along the way.
Blow away your local database (and with it your __MigrationHistory table, and it’s state)
Delete all of your local migrations.
Pull the latest trunk/master Domain Model from your Source Control repo up into your branch.
Merge the latest Domain Model with your local changes.
Add single migration of “the difference” between the repo and your domain model.
Test a migration “up” on your local DB.
Merge your branch back into your mainline.

Summary

The main differences between these two workflows is that the first method saves you a little bit of time between small domain model changes. This can be really handy if you have remote teams where some members are working together in the same room on the same branch/head, as it reduces a bit of friction while they are working. This also tries to bring the problem back to being just what it should be: a code merging one with your domain classes.

I am still a big fan of Code First Migrations, and when you work with others in a team that uses them you can make some huge performance gains if you take the time to make your human processes play nicely with them. With a bit of practice the pay off is definitely worth it.

SimpleDeploy.Net - An FTP Style Wrapper for the MS Deploy API

1/5/2013 11:36:59 PM

MS Deploy is such a powerful tool when used to keep your applications and services up to date. What is even more awesome is that the API for MS Deploy is available for you to write applications that utilise this power from within your own applications. As most people who’ve played with MS Deploy can report though, whether you’re using MSBUILD, PowerShell or the .Net API; The MS Deploy API sucks when it comes to simplicity. So after trial and error and much head banging I’ve created a wrapper for MS Deploy’s API to help you complete simple tasks with less friction.

I've spent a fair bit of my personal time working on a personal project that I'll be releasing in the next few months. It involves the use of the MS Deploy API for deployment. I’ve had to get very close to the API that MS Deploy makes available, and in doing so have come to have a love/hate relationship with how it has been written.

MS Deploy contains a very powerful toolset but file upload and download is definitely only the tip of iceberg of this functionality. This is my problem. The team who wrote the API appears to have been so involved with the bigger problems, that the act of simply uploading a file to a remote server is a painfully long winded and hard to understand task:

Ms Deploy API Upload Example:

var destinationOptions = new DeploymentBaseOptions
{
    ComputerName = "https://myserver.com:8172/msdeploy.axd?site=[MyIISWebSiteName]",
    UserName = "myUsername",
    Password = "myOoberSecurePassword",
    UseDelegation = true,
    AuthenticationType = "Basic"
};

var syncOptions = new DeploymentSyncOptions { DoNotDelete = true };

var remotePath = String.Format("{0}/{1}", "[MyIISWebsiteName", "theRemoteFilePathToSaveTo.txt");

using (var localFile = DeploymentManager.CreateObject(DeploymentWellKnownProvider.ContentPath, @"C:\temp\fileToUpload.txt"))
{
    var result = localFile.SyncTo(DeploymentWellKnownProvider.ContentPath, remotePath, destinationOptions, syncOptions);
    //if (result.ObjectsAdded == 1 || result.ObjectsUpdated == 1) everything has worked
}

Welcome SimpleDeploy.Net

So I took the liberty of making it easier for people like me who just want to freekin-upload-some-files. I haven’t looked back.

SimplyDeploy.Net is a very simply library with 4 simple methods:

public DeploymentObjectList FetchFileList(ConnectionProperties properties)
public bool DownloadFile(ConnectionProperties properties, string relativeRemoteFileName, string fullLocalFilePathToSave)
public bool UploadFile(ConnectionProperties properties, string relativeDestinationFileName, string fullLocalFilePath)
public bool DeleteFile(ConnectionProperties properties, string relativeRemoteFileName)

It uses the ContentPath WebDeploy provider to make all of its remote calls in keeping with Visual Studio 2012’s Web Deploy Publishing Profile support. Hopefully this will allow for the most coverage from servers/sites that already support this.

Getting it

It is released under MS-PL so is fine for commercial use.

Either download the source from GitHub - https://github.com/dougrathbone/simpledeploynet

Or install using NuGet:

Mobile, Tablet and Desktop Development All at The Same Time

1/16/2013 3:55:43 PM

When you go to create a mobile app for each major platform you quickly realise that it’s a mind boggling task with many languages and tools out there to learn along the way. A number of products have arrived over the last few years that enable you the freedom of only having to care about one language and toolset, with the dream being that they take care of the rest. Does this type of approach work? DXTREME does a pretty good job of making the answer to this question Yes.

A recent product I've been playing with is called DXTREME from DevExpress. For US$899, it sells itself as enabling you to create hybrid apps for iOS, Android, Windows 8 and the web all in one toolset with a single codebase. While this is not necessarily something new, DevExpress does more than just cross compilation taking it further by also augmenting your app’s design so that you can create native looking apps for each platform without having to do so much heavy lifting. One codebase, but your iOS app look like a cocoa-touch interface, and your Android app follows similar modern droid device specific design patterns.

Developing for the Web, when it’s not developing for the Web.

When I first set out on my cross-platform adventure, I did a lot of reading on the DXTREME tooling from DevExpress. I soon came to realise that it is really a framework and tooling package for writing mobile apps by wrapping itself around Apache Cordova (PhoneGap), and then adding Visual Studio support for build, design and emulation. And this is a good thing. PhoneGap has been around for a considerable amount of time. It’s stable and accepted as a robust solution for mobile development. What has been extremely lacking with PhoneGap on Windows to date is that there hasn’t really been any good IDE support for it – and this is where DXTREME knocks it out of the park. So much so that even if it did nothing else I’d buy it just for this: it adds PhoneGap support to Visual Studio, including when using Visual Studio’s Designer and Script Debugging features.

Because the apps that DXTREME produces are really hybrid HTML and JavaScript apps, in essence you’re writing a single page HTML/JavaScript application (SPA). I’d hazard a guess that these two languages are the most well known in the world, making it easy for developers of all skill levels to get in and have a go at app development. Like Microsoft did with ASP.Net MVC, DevExpress have also used technology that is already commonplace in your web development toolbox, with the apps created by DXTREME using a number of common JavaScript frameworks like jQuery, Knockout and jQuery Globalize.

Another area where DXTREME stops being just PhoneGap is where, as I mentioned above, it adds a hell of a lot of custom CSS and JavaScript magic to make your app’s view pages look platform specific so that your users will never know that you didn’t write separate applications for each platform. It does all of this without you lifting a finger – this in itself is a very nifty trick.

An example page and it’s code:

<div data-dx-role="view" data-dx-name="Index" data-dx-title="Home">
    <div data-dx-target-placeholder="content">
        <p style="padding: 5px;">
            My First DXTREME mobile app.       
        </p>
        <p style="padding: 5px;">
            Cross device development is off-the-chain!
        </p>
    </div>
</div>

What this looks like when exported to iPhone, iPad, Android Phone, and Microsoft Surface. Note the platform specific look and feel.

On top of the cross device design features. The built in Visual Studio Emulator that the above screens have come from supports an impressive list of devices to help you get a better feel for how your app will look:

iPhone 4
iPhone 5
iPad
iPad mini
iPad 3
Nexus Galaxy
Galaxy Tab
Nexus 7
MS Surface

You have to admit, this is a huge list of emulation devices when you’re only having to maintain a single codebase.

Hello World.

Taking a look at when you first fire up Visual Studio after installing DXTREME, if you go to create a new project you are met with a few new project templates:

The first two are cross platform applications, the first being a basic template that has a single project that cross compiles to multiple devices and the second is a template that helps you scaffold your application based on an OData domain model from a remote (or local) web service.

This is an interesting choice of approach for a scaffolding data source, and when I first played with it I tested with the Stack Exchange public OData sources as I didn’t have any OData services on my speed dial list. Although I understand that OData is a great way to plug in a web-accessible data source, the fact that there is no other data source supported by the scaffolding tool feels a little limiting, especially if you require any form of more advanced authentication support for your remote data.

The second two templates are actually the solution to the above short-coming in that they are scaffolding project templates for WCF web services that allow you to serve out OData. These templates themselves are scaffolded from an Entity Framework data source. This means that even if you don't have an OData service today, the tooling tries hard to make sure this doesn’t slow you down too much.

Getting Answers

When you learn any new tool or framework I often get a little worried when it comes to how fast you can get up and writing application code. While DXTREME is off to a good start with the very fact that they use HTML and JavaScript for their language support, the auto-magical cross-device CSS and JavaScript components where still new to me. It worried me that I would be spending time learning (banging my head against a wall?) an abstraction when I really could be learning the real thing by spending time using Objective-C or Java – but I was in luck because there appears to a fair bit of official documentation for their framework items.

The rest of the components used are in their original states. jQuery is the real thing, using Knockout for MVVM model binding is also the exact same as can be downloaded from their website. You can also upgrade these files out-of-band by downloading the latest versions of them, although one feature that I missed was being able to use Nuget to add Knockout and jQuery as packages for easy upgrading (Visual Studio wouldn’t let me open the package manager for the DXTREME project type) – I hope they add this support in a later version.

There are documentation posts and examples of using jQuery and Knockout literally everywhere over the internet; This was a great move in choosing to use common frameworks.

The same goes for any of the device specific hardware calls, as these are completed by the PhoneGap/Apache Cordova wrapper so documentation can also found easily all over the place.

Hardware Interaction

Modern smart phones often require you to get a bit closer to the metal when you’re planning on writing code that talks to the hardware on the device. Maybe you want to use the device camera to take a photo, or ask the location services on the device for your current GPS co-ordinates. How do you do this when you’re just writing an HTML app though? HTML and JavaScript don’t have support for hardware interaction at this level yet, and device specific APIs for iPhone, Android and Surface are all different, so how does a DXTREME app allow you to take a photo?

Luckily the heavy lifting is done elsewhere thanks to the PhoneGap framework which has had long running support for this already:

function takePhoto() {
    navigator.camera.getPicture(onSuccess, onFail, { quality: 50 });
}

function onSuccess(imageData) {
    var image = document.getElementById('photo');
    image.src = "data:image/jpeg;base64," + imageData;
}

function onFail(message) {
    alert('Failed because: ' + message);
}

Local storage is taken care of by a similar augmented JavaScript connection to the database technology SQLite, which runs the only slightly cut down SQLite SQL spec:

var db = window.openDatabase("Database", "1.0", "My App", 200000);
function queryDB(tx) {
    db.executeSql('SELECT * FROM MYCONTACTSTABLE', [], querySuccess, errorCB);
}

function querySuccess(tx, results) {
    alert("Contacts found in DB: " + results.rows.length);
}

Of all the hardware devices that DXTREME supports through use of PhoneGap, the only thing that lets this down a little is support for these technologies in the Emulator. It would be great for there to be support for each of the device hardware wrappers inside the emulator to make it easier to test and debug without deploying to a device. Being JavaScript this can easily been bootstrapped for testing.

Squashing Bugs

While talking about testing and debugging hardware it’s worth taking a look at how debugging works for a DXTREME app, as they’ve created a pretty innovative approach to this mundane task.

Debugging a DXTREME app works very similarly to a normal Single Page Application you develop for the Web. You start your app inside Visual Studio, which fires up your favourite browser, but instead of looking at your app directly, it gives you a very cool abstracted emulator view that allows you to switch between devices (and therefore DXTREME hybrid views):

Navigating your app, and debugging it’s JavaScript operates just as if you were debugging a JavaScript app. As I use a combination of Firefox or Chrome’s debugging tools this works perfectly for me, but if you use Internet Explorer this means that you have full support for the Visual Studio Script debugging.

Another interesting thing to note is if you look at my screen shots you’ll see a QR Code in the emulator (bottom left):

If you scan this with your device you’ll be able to play with the app using a slick Reverse Proxy technology that routes requests to your local IDE/dev environment through DevExpress’s website, and although you can’t test device specific functions such as camera and location services this way, if you download the iPhone “Courier” app from DevExpress, you can test all of this native functionality in the same way by scanning the QR Code (apparently an Android app is on the way soon, no word on Surface or Windows Phone support yet).

Getting it on a Device

Once you’ve developed your app, whittled away your bugs, and are ready to actually try your app on a device DXTREME also appears to have you covered with Integrated Application packaging built into Visual Studio. From your application’s properties page you can assign App icons, add your Android signing certificate and iOS Publishing Profile and actually build an iOS IPA or Android APK package.

This blew me away. In the past I didn’t know there was any way to build iOS apps on Windows. Regrettably you still can’t actually test publish your app to an iOS device without a Mac, but as with many Apple development traditions, this is an Apple-dictated hurdle rather than a DXTREME shortcoming.

Also, because you are really developing an app that sits on top of PhoneGap, you can also leverage a service like PhoneGap Build to take some of the hassle out of bundling your application. For Android this can be handy: PhoneGap Build allows you to download the app onto your device from the Web once it's finished."

In the End

If I was to dream up my own cross-platform development tool tomorrow it wouldn’t be anywhere near as good, or as feature-rich, as the product that DevExpress has created in DXTREME. The thoughtful way in which the framework has been designed to use common languages and JS frameworks makes it a breeze to pick up, and the heavy Visual Studio integration alone makes it worth looking at purchasing just to help speed up any current PhoneGap development you might be doing.

When developing cross platform apps learning a hybrid development framework raises questions: Does this approach really work? Does it actually make you more productive? Is it worth learning a language that is native? After spending a few weeks with DXTREME, I believe strongly my answer to these questions is a resounding Yes. As it’s easy for you to try it for yourself, why not go check out the free trial?

I received the product mentioned above for free in the hope that I would mention it on my blog. Regardless of this, I only recommend products or services I use personally and believe my readers will enjoy. I am disclosing this in accordance with the Federal Trade Commission's 16 CFR, Part 255: "Guides Concerning the Use of Endorsements and Testimonials in Advertising."

Why Nuget Package Restore’s Not For Me

1/31/2013 5:41:14 AM

Nuget has become such a valuable part of the .Net ecosystem it's any wonder how we got the job done with 3rd party packages without it. When working on projects in a team many developers turn on Nuget Package Restore to save them having to check their packages into Source Control. This allows them to have their packages download whenever a new developer goes to build. It’s also quite popular with project teams that have Continuous Integration setup. I recommend against Nuget Package Restore, as I’m simply not a fan.

"Awesome Sauce on Every Build"

When Nuget was first released the majority developers in .Net land rejoiced. No longer would we have to download, checkout, test and build third party open source libraries. We also didn’t have to worry about keeping them up to date. More importantly it also put a number of .Net developer’s minds at ease, as the next time they struck up conversation with someone they knew from the land of Ruby on Rails they finally had a response when the topic turned to discussions about their mystical "Gems"!

Nuget Package Restore soon followed Nuget and allowed us to easily have our packages downloaded on every build without having to store your package binaries in your source control. This makes it easier to share projects with fellow developers as you only need to send your code. With Nuget Package Restore your build server can also happily "fill in the blanks" for any packages that it was missing every time it went to build a new release for you.

I argue that not all of this glittery stuff that comes with Nuget Package Restore is gold, and I actually think it teaches bad habits and causes any number of issues that didn’t exist before it graced our developer bat belts. As I haven’t heard anyone raise much comment in line with my line of thinking I thought I'd vent the only way a guy from the internet like me can. A whingey Blog post.

Faster Builds

The first and simplest reason I’m not a fan of Nuget Package Restore is how it affects build times in teams where you’re using Continuous Integration (i.e really any team that understands the value that CI brings; if you haven’t taken a look, now is the time). When using a build server, large projects take time to finish building. Your build server has to download your source, build your binaries, run your unit tests, any integration tests and optionally deploy your application. Every second counts.

A large solution may take many minutes to finish building and testing. If your build server has to download packages on every build you’re simply stealing valuable time away from yourself, as you are just extending the time it takes to have your unit tests fail, and therefore give you less time to troubleshoot any bugs you’ve introduced from your last check in. By removing the need to download your Nuget Packages you instantly speed up your build to the breakneck speed of…

Wait for it…

The speed you had in the dark old ages you checked in your binaries (shock horror!).

The only time adding binaries to your source control doesn’t speed you up dramatically is in certain situations where you are using DVCS like as Git or Mercurial. Over time it may slow down your clones on large repositories. But like most things, this is just an excuse to not dig deeper. There are ways to work around these issues by using techniques such as using Git submodules to store binaries in another repo.

Reliability

This is actually my number one reason for steering clear of Nuget Package Restore. If Nuget.org is down, your builds are down.

“… As if that’ll ever happen, the gains are worth the risk…”

But my opinion is you’d be wrong. Case in point is the fact that there has been 2 periods where Nuget.org has been down for a few hours each just this month alone (January 2012):

and it gets worse:

So if I interpret this for you: If you share your project with anyone or try and do an automated build on your build server and Nuget.org is down, you’re fresh out of luck!

Peter Newhook summarises this well:

This isn’t a reflection of Nuget.org – we don’t pay for the service. After I experienced this for the first time, I realised that using Nuget.org as a build dependency wasn’t something I could put my team or clients up against. And the solution was simple: I checked in my Nuget packages, avoided Nuget Package Restore and never thought about it again.

Control Over Dependency Versioning

Another thing with Nuget Package Restore that I’m not the biggest fan of is the way it effects your ability to ensure your QA teams they don't need to take another look after a build.

When you’re working with a set of binaries on your local environment and then check your code in without the packages you were using, my gut feeling is that there is no absolute way of knowing that the very same binaries will be used by your team mates or the packages downloaded by your continuous integration environment. You are ceding control of this to the Nuget gods. If one day Nuget just happened to serve a different package to your team mate or build server bugs may be introduced. At this point you've unnecessarily introduced a roadblock.

Like my other complaints, this problem is one that’s easily avoided if you just check your packages in, and turn off Nuget package restore.

Ask yourself, what happens if the next time you open an old project and you no longer can download the package because it’s been removed from Nuget? Or if there is a technical difficulty at Nuget that stops you from getting it.

Laws of Continuous Integration

This brings me to my final and probably most passionate reason for wanting to avoid Nuget Package Restore: I feel that having your solution depend on remote dependencies breaks one of the elementary rules of Continuous Integration/Deployment and Source Control usage. To me this rule is one of the most important rules I try to live and die by as a developer:

“… Source control is number one in both configuration and logic storage. A new developer needs to be able to have everything in a state where they can load it up in 1 command, and deploy it with another…”

-Evan Bottcher

Living by this thinking dictates that what is in your source control is more than just your code. Your application is code, configurations, dependencies, and anything needed to build your application. I’ve heard some people even say that they suggest placing their IDE and any tooling they use in their source control. This kind of thinking definitely includes your packages. If a new team member can’t get the folder for your solution and be ready to roll, I get the feeling I’ve divided by zero with the universe. This may just be a personal thing for me, but it feels pretty sacrilegious to do it any other way.

Rewind the Clock, and Check-in

If you’re like me and consider any of the above to not sit right with you, and any of the annoyances or inconveniences that using Nuget package Restore can bring along for the ride when you use it doesn’t feel like it’s worth it, then I’d suggest you follow my lead and avoid using Nuget Package Restore in a production environment. Check in your packages, it’s that easy.

Update 3rd February 2013.

I’ve received a fair few messages on Twitter about this post. So I thought I’d take the time to clarify a few things:

This post is about Nuget Package Restore – not Nuget.
I love Nuget, and have very little else to say about it; even though Nuget does go down from time to time, it’s acceptable. On the other hand, I strongly feel that you should be checking in your packages instead of having Nuget Package Restore download them on your build server or when new developers start using your project.
I understand there are a few ways you can make the situation a little better. This doesn’t change my view.
You can install a package server locally. You could cache or backup your Nuget cache folder. None of these solve the biggest issue I have in that you don’t have everything your project needs to build included in Source Control.
I mention that the speed on your build is important, and that Nuget Package Restore slows this down slightly. I didn’t back this up with data – If I wanted to make a big point out of it I should have. This wasn’t my main concern though. I mentioned you’d get faster builds if you check in your packages. This should be the case if you have 1 package, 10 packages, slow internet, fast internet etc – so I didn’t really feel the need to go into too much detail. If I have more time I will take the time to write about it soon.
As with everything I blog about, this post is my opinion. You may feel differently about the topic. This is OK. It doesn’t mean that I’m saying that “If you use Nuget Package Restore, you’re an idiot”, but it does mean that I think you might either not be fully aware of what using Nuget Package Restore means for your project, or you do and are OK with it. Keep on!

All Bugs Are Not Created Equal

2/7/2013 5:30:30 AM

You’re close to shipping and you receive a shopping list of bugs and changes. Some are tiny and un-eventful, some are show stoppers, some let the bad guys in, and some are simply scope creep trying to sneak through the door. It’s hard to know where to start without reclassifying them because the majority of them are all labelled Critical. It’s time to sit down with whoever documented your bugs and do some talking…

Listening to The Little Things

I’ve got a secret to share that might make you snicker at me: I used to be a DJ. I played in clubs, and spent large amounts of time and money on music. I still have gear at home to keep the dream alive whenever I feel like playing some Electro.

One of the things you might not have thought about when it comes to DJ’ing is what it’s like to be in a pumping club in the booth.

You have all of the club’s sound system in front of you playing towards the back wall. Loudly.

You have your booth monitor (speaker) turned up louder than the club music so you can hear it “in time” without you mistakenly thinking it’s playing 1 second later because it’s bounced off the back wall.

You also have your head phones on with the next song playing in time, again slightly louder than the booth monitor.

It gets loud.

As the night goes on and your ears get tired, you fight yourself to not turn everything up louder so that you can still hear everything clearly. This is why DJ’s commonly get Tinnitus – everything gets so loud that you struggle to tell the difference between different sounds as you start to go deaf.

Badly prioritised Bugs have the same effect.

Signal/Noise Ratio

Humans are amazing at prioritising if given relevant context information, however when we’re under pressure both project managers, clients and developers alike often fall into the trap of miss-prioritisation with the end result being a wall of “must have” fixes.

When looking back on these situations it’s understandable to see why some of us respond this way. We’re often dealing with a set of not so good circumstances such as:

The application needs to go into production a day from now.
There are still things you’re working with your testers on that are fundamental. Maybe you don’t even have testers…
Maybe the design or BA team decided a week ago that part of the site needs to change in functionality before it ships.
But: Everything still needs to be complete in around 24 hours.

In the above situation anyone without any formal guidance, training or experience may easily class “The menu has to be blue before launch” at the same priority as “there is a bug that stop some users from completing the checkout process”.

“…I mean; you understand. This must be completed before we go live…”

To most of us who’ve been here before this seems ridiculous. But in hindsight these situations always look 20/20 – experience simply means you’ve had this 20/20 vision a few times before.

Whatever side of these situations you sit on, classifying bugs and any of the stress or lack of clarity around doing so subsides if your team agrees up front to classifications for bugs.

Get Your Priorities Straight

Every company and software product is different and because of this you should look within your own team and work to not only define the labels for bugs, but how you come to these decisions. Something everyone can agree on.

A common way of doing this is called Cost Of Delay (COD for short), and basically it outlines a framework to asses how important a bug is to the business’s bottom line.

If I use my original example of “The menu has to be blue before launch, a web design agency that has a strong design focus probably has a much higher COD for implementing such a fix than a back-of-office internal developer team. They sell a service of being pixel and colour perfect every time. For context though the same web design agency still probably classify this bug as having a lower COD than “the site doesn’t load at all on a major browser”. It’s not rocket science, but when you think about it using this approach it takes the emotion away. Your designer should understand this just as well as your CEO.

Another good place to look for inspiration on how you classify bugs in your team is by looking towards what others are doing in this space.

If you want to see how some of the big guys do it, take a look at Bugzilla. This system take care of bugs for Firefox, the Linux Kernel, Gnome, Apache and Open Office.

While they classify bugs into similar simple categories, they also work hard to define how they come about these decisions – this kind of straight forward classification should be agreed up front in every team before you start a project.

Severity	Meaning
Blocker	Blocks further development and/or testing work.
Critical	Crashes, loss of data (internally, not your edit preview!) in a widely used and important component.
Major	Major loss of function in an important area.
Normal	Default/average.
Minor	Minor loss of function, or other problem that does not affect many people or where an easy workaround is present.
Trivial	Cosmetic problem like misspelled words or misaligned text which does not really cause problems.
Enhancement	Request for a new feature or change in functionality for an existing feature.

I Doug, do solemnly swear…

Before my next project kicks off I’m going to set out a standard for how my team plans to classify bugs – and more importantly agree on a definition on what falls into these classifications.

When next entering bugs into your bug tracking system stop and ask yourself “Is it stopping my application working?”, “How does this compare to other blocking bugs” and other reasonable questions before hitting submit. Work to keep your bug list’s Signal/Noise ratio in a state where everyone can still prioritise correctly.

Your other team members will all thank you for it – Probably because they’ll all be able to still hear at the end of the project.

Get Your Thinking Caps On –# AppFest Sydney

2/11/2013 5:19:36 AM

This weekend a bunch of like minded people are getting together to hack on Windows 8 and Windows Phone to create some awesome applications and win sweet sweet prizes. You can be one of them – so why don’t you come rock out with Microsoft, eat and drink on the house and create some app awesomeness.

Price: Free

When: 9:00am – 9:00pm on Saturday 16 February 2013 and 10:00am – 5:00pm on Sunday 17 February 2012

Where: UNSW Kensington Campus – Scientia Conference & Events Centre in the Tyree Room

The plan for the weekend:

Saturday 16th February

09:30 Doors open / Registration

10:00 Welcome

10:15 Design Keynote – Shane Morris, Automatic Studios

11:00 Hacking and Design prototyping

13:00 Lunch

18:00 Dinner

21:00 Doors close

Sunday 17th February

09:00 Doors open

12:00 BBQ Lunch

13:00 Demonstrators prepare

14:30 App demos

16:00 Success keynote – Vaughan Knight, Nokia

16:30 Prizes announced

17:00 Doors close

As a Windows Phone developer I’m really excited to see what apps come from this weekends inspiration and hard work. If you’re a start up, have an idea but no idea on how to create it or just want to come check it out register and come along.

I have the privilege of being a Microsoft Invited mentor over the weekend and will be wandering around helping out where I can, so If you see me please come say hi.

Music to Your Ears - Fixing the Win 8 Music App

2/19/2013 6:22:29 AM

Windows 8 comes with some pretty awesome new toys, one of these for me is the Music app. As a Windows phone user for many years now I’ve fallen in love with the all-you-can-eat music subscription service that Microsoft has on offer. Sadly for me when I installed Windows 8 the app simply crashed every time I tried to play music. I searched for months and months, and now have the answer – in the hope that I save someone else the heart ache I’m happy to share what I discovered.

It appears that the very issue has a lot to do with your audio drivers and whether they are signed or not. Microsoft assumes that if the drivers are unsigned, then protected audio or music and video that has been protected with DRM should not be playable – you might be an evil doer after all.

When Windows 8 came out to developers last August I backed everything up and jumped into the deep end. I have a Lenovo x220 and at the time only beta drivers were available – I took the risk, and flew in blind. After going around the world and back it appears that these drivers and more importantly the fact that they were unsigned, meant that I simply couldn’t play or watch content from Xbox Music.

The Error

Can’t play.
Please try again. If the problem continues, visitor www.xbox.com/support to check for guidance.
0xc00d11cd (0x8000ffff)

It was driving me up the wall. Searches of Google simply showed many happy people talking about how awesome Xbox Music on Windows 8 was…

oh, the irony…

Solution Time

Then one day I was reading a post that someone wrote about issues they were facing with Netflix and Windows 8. The Netflix support staff mentioned something about the DRM features in Windows 8. Something clicked in my mind.

Some more Googling and a few minutes later I came across this post.

The important part:

Open regedit and navigate toHKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Audio.
Change the DisableProtectedAudioDG value from 1 to 0 and reboot.

So now’s your chance. Follow the bouncing ball and retrieve you DRM-less glory, however as you are making a change to your registry I highly recommend that you back up your registry before implementing the change.

Happy content consumption.

Preparing Your Windows Phone 8 App For Submission to the Windows Phone Store

2/26/2013 5:55:26 AM

Finishing your Windows Phone 8 application isn’t the end of your Windows Phone journey. You’ve now got to get it into everyone’s hot little hands – by submitting it to the Store. Having developed for the Windows Phone platform now for a number of years, I can attest that just uploading your XAP file isn’t the only thing required to make this happen. Similar to my post on “Must have tools for Windows Phone” I'll try and fill you in on a few things I wish I'd known before I submitted.

Last weekend I was lucky enough to have been asked to spend the weekend acting as a mentor for Microsoft’s #AppFest Windows Phone and Windows 8 hack weekend. As developers came closer to finishing the first version of their apps, many started asking the same thing:

What do I need to know next?

So below I’ll fill you in on things you need to get ready, maybe do ahead of time and generally start running through your head before you push your XAP file to Microsoft.

Get a Developer Account

The first thing I’d do right now, is register for the Store.

It seems obvious, but doing this ahead of time will help you understand any further requirements, and give you a feel of what's to come. It’ll also help you prepare your US tax status if you want to charge cold hard cash for your app. Microsoft doesn't need a US ITIN if you’re outside the US anymore but you still need to submit tax info. If you’re wanting to submit under a business name that doesn’t yet exist, you’ll need to get this information on the way as well.

You can check out the “getting paid” page on MSDN for more info. They say if you’re a US citizen you’ll need to provide either:

Social Security Number (SSN): 9 numbers in the format of 123-45-6789

Employer Identification Number (EIN): 9 numbers in the format of 12-3456789

Also, it’s worth noting that only the following countries’ bank accounts are supported to receive payouts. If you’re outside this list, you might want to rethink your monetisation strategy – maybe ads instead?

Review the UX guidelines

There are a few UX guidelines Microsoft has prepared for the Store that your app must adhere to before it’ll get approved. Knowing these ahead of time could save you some serious time and frustration.

The guide: http://tinyurl.com/wpstoreguidelines

What this means in “human terms”:

Backstack navigation - the back button must exit your application when it first loads. Elsewhere, it must always go to the last visible page. If you’re redirecting to a “login” page or settings page on first load, remember to remove it from the backstack.
You must allow users to turn off location services – if you use location information on load, throw a message box asking for permission before doing so.
Always ask users for permission before turning on toast notifications. Don’t simply set the default to “on” and think having a settings page to turn them off is enough.
If your app plays audio, it must be able to play non-step for more than 6 hours. Use the Visual Studio Store Performance Tools to verify the battery usage.
Make sure you test you app using the 720p emulator to optimise the viewing experience for everyone.

Contact Information

You need to have a support page on the web for your app and a valid email address or about section in your app. Microsoft test this web page during app testing and will fail your app if the page, the version number and “a way to contact you” isn’t listed. I know first hand because my app InTheKnow failed once because of this.

You must list your app’s version somewhere in your app. Think about an “About” page.
You must list a way to get in contact with you. Again an “About” page will fix this.

The venerable Jeff Wilcox has written a great about page helper blog post to get you started.

App Description

Start thinking about how to describe your app in the Store. Keep it simple and to the point. Metro guru Shane Morris talked about defining your app’s “best app statement” at #appfest, and this summarises it well.

"my app is the best app for..."

Make sure to mention things that set your app apart or are Windows Phone specific advantages.

Take a surf of the Store and start having a think about what category you think your app sits in – observe how similar apps are priced, and whether they support “Trial Mode”.

Your App’s Art

If you want your app to be a success, you’ve got to leave people with a good impression from the moment you open your app.

Start towards this by ensuring you have a decent loading screen and icon.

If you’re looking for inspiration check out: http://thenounproject.com. They have a metro style icon for everything.

Take some interesting screen shots of your app within the emulator. Make it look “real” - have a think about the sample data you enter in your app before taking screen shots to make them more appealing. A ToDo app looks just like every other ToDo app unless the screen shots show content that is interesting.

Also, remember to take extra screenshots for 720p and 400x800 screens.

Run The Store Test Kit

From Windows Phone 7.1 onwards Microsoft has included a great little store test kit full with automated tests to run against your app before you submit it. A large number of these tests are the same as those used by the Store testers when reviewing your app. To run these tests, simply right click your Windows Phone 8 project and select "Open Store Test Kit". Don’t proceed with submitting your app without running this tool.

Consider also using the manual profiling tools included to ensure your app doesn't "do evil".

XAP File Submission

Before you hit submit, make sure the XAP file you submit is a RELEASE build. Sounds simple, but it’s pretty critical.

Also consider obfuscating your app using Dotfuscator Community Edition. At the end of the day, your XAP is a zip file of your binaries. The security of your codebase on the Windows Phone platform has been known to not be airtight (I blogged about this last year) and while I haven’t looked at the situation on the Windows Phone 8 platform yet, you can never be too safe. So do what you can and at least make it harder to extract your source code if anyone gets a hold of your XAP.

Let 'er rip.

Once you’ve run through the above, you should be ready.

After submitting apps for myself and friends, I know the excitement that builds as you press play on something you’ve been working hard on for days/weeks/months – good luck!

Testing the Untestable with Microsoft Fakes and Visual Studio 2012 Update 2

4/10/2013 6:42:10 AM

When maintaining applications built in ways that make unit testing them high friction, difficult or down right impossible, we often turn to integration tests or no tests at all. Poking around the outside of your logic to see that the outcomes our code produces is often fiddly, brittle and in most cases so time consuming it just doesn’t happen. Lucky for us all, if there has been a case in the past where we’ve been unable to test part of our business logic, Visual Studio 2012 Update 2 is here to offer an answer to your prayers which you just might not be aware of.

Recently I changed jobs from working for external clients on creative driven work to running an internal development team. This has been the other side of the universe in more than one way for sure.

With this change in scenery, so changed the types of projects I’ve been working on, with a change to more brownfield work. Out with the new and in with the old… or something.

My old role consisted of about 10% legacy project and 90% new. Often the 10% brownfield projects were themselves only less than 2 years old. A common attitude this bread within my team whenever they had to go back to an older project was something along the lines of:

“…

Oh No!

You mean I have to work in .Net 3.5…!

”

Us poor Gen-Y developer equivalents…

My new role, and this change to supporting a number of older applications has brought with it a need to look at ways of addressing untested, untestable or black box legacy code to increase our confidence in refactoring it.

The Problem

Writing unit testable code has been an ever growing trend in .Net, but 5-6 years ago it was still growing mass acceptance in our part of the world – partially lead by Microsoft themselves; Visual Studio 2008 Pro was the first IDE to have unit test support built in outside of the “team suite”, “ultimate” or “architect” editions. This gave the impression that “Unit tests were only for the rich”.

What I’ve seen in many workplaces over the years is that this lack of unit test penetration lead to a lot of .Net developers simply not learning good practices when it comes to separation of concerns and modular coding practices that lead to testable code. This isn’t a complaint. We were all there once; just a statement that I saw the effects of this in a lot of legacy projects that all suffered from:

Wide spread usage of static classes and methods (i.e. no interfaces, virtual methods etc).
A lack of dependency injection (no way to easily replace functionality with a mock or a stub).
Tightly coupled code (a domain class that talks to the database, or a business layer method that refers to HttpContext.

This lead to scenarios that meant that when developers on .Net started out unit testing, they often were forced into doing end-to-end integration tests to get around their tightly coupled architecture.

[TestMethod]
public void UserSignup_ValidFormSubmission_UserIsCreated()
{
    var testHelper = loadApplicationOverHttp(Configuration.LocalSiteUrl);
    testHelper.SubmitSignupDataOverHttp("username", "password");

    var dbEntry = DataLayer.GetUser("username");

    Assert.IsNotNull(dbEntry, "No user was created in the database");
}

From a build and automation perspective, this type of testing is expensive as it takes a long time to build and run tests. To start with you need to have a database server online just to run your tests. You often need a clean database. It needs to be kept in synch with production or a known state. When working with CI it’s harder to automate, and hell when you’re just a new developer on the team you’ve got to work hard to get your workspace into a happy place to even run these kinds of integration tests.

This lack of smooth experience only leads to one thing: less testing.

While Integration tests are useful to know that your application works end-to-end, obviously the fiddly and brittle nature of them is why Unit tests are such a preferred method of testing code path functionality across your entire codebase. But Unit Testing takes architecture choices from day 1 to make your applications easily testable, so brownfield projects caught out with a lack of good architecture with little budget for refactoring work often flounder in a unit testing no mans land.

That was until Microsoft Moles came along.

With Moles came a huge power of interception that was originally a free piece of framework/tooling from Microsoft Research that worked with Visual Studio 2010, albeit lacking some polish.

Sadly Microsoft took Moles away from us with the launch of Visual Studio 2012. In a similar turn to how they approached VS 2005 they opted to only include support for Moles in the Ultimate edition.

That was until the Update 2 for Visual Studio 2012 released last week, which if you download today has all of the ~~moles~~ fakes magic (they changed the name), but now with Visual Studio 2012 support and awesome IDE integration.

Microsoft Moles/Fakes

One of the greatest things about unit testing in other more dynamic languages like Ruby or Python is that testing logic is inherently easier because you can simply replace functionality at runtime as they’re dynamic languages. Mocking is almost a non-event just because of this dynamic nature.

Thanks to Fakes some of this power is yours to tap into with VS 2012 just as it was when they first released Moles in 2010.

Moles has grown into what we know now as Microsoft Fakes and is an incredibly powerful bit of kit for allowing you to replace functionality for your application at runtime. This power then allows you to intercept legacy code and make it testable.

A good example of this power is in the example below of a shim of System.DateTime.Now and its Get accessor. Microsoft Fakes allows me to test a static method or anything that depends on it with ease; in this case changing the date to 1st January 1970:

[TestMethod]
public void MyDateProvider_ShouldReturnDateAsJanFirst1970()
{
    using (Microsoft.QualityTools.Testing.Fakes.ShimsContext.Create())
    {
        // Arrange:
        System.Fakes.ShimDateTime.NowGet = () => new DateTime(1970, 1, 1);

        //Act:
        var curentDate = MyDateTimeProvider.GetTheCurrentDate();

        //Assert:
        Assert.IsTrue(curentDate == new DateTime(1970, 1, 1));
    }
}
public class MyDateTimeProvider
{
    public static DateTime GetTheCurrentDate()
    {
        return DateTime.Now;
    }
}

Pretty cool!?

And to get this level of interception all you need to do to get this is download Visual Studio 2012 Update 2, then Right click on a Reference in your Unit Test Project and add a Fakes Assembly (a shimmed copy of your binary).

Then all you need to do is:

Wrap your logic in a ShimsContext.
Use the new convention based syntax “Shim[Classname]” class and “PropertyName[Get/Set]” syntax to define what your shim will do.
Go grab a <insert tasty beverage of your choosing/>

And you’re off and away!

Stop reading – Start testing the untestable!

With the power of Microsoft Fakes offered in Visual Studio 2012 Update 2 there is now no excuse not to test that old .Net project you and your team members are afraid to refactor. Get to it.

MSDN: http://msdn.microsoft.com/en-au/library/hh549175.aspx

Introducing My Cloud Deployment Service, OnCheckin.com

5/21/2013 7:07:52 AM

Constantly over the past few years I’ve thanked my lucky stars that I’ve had continuous integration and deployment setup for the web sites I’ve been working on. It is one of the few development opinions where I break the “religion and politics” laws of social etiquette. I feel incredibly strongly about the benefits that having a good Continuous Integration and Deployment story can bring to a project and it’s team. Sadly, not everyone has been able to experience the awesome sauce of CI/CD. I’ve been working hard to change that.

Teams using Continuous Deployment know that the cost savings, safety and reliability that comes with deploying many times before going into production are priceless. Awesomeness aside those of us lucky enough to be putting it into action daily all found out quite early on that unless you’ve done it many times before, starting out with continuous integration can appear to add overhead to your timelines that you may simply not have time for.

I’ve used many different approaches over the years, I’ve setup Continuous Deployment starter packs for others to use to get started, I’ve blogged about it many times to try and spread the good word, but it still felt like the level of friction for newcomers was too high.

After all, you need to;

Setup a build server such as TeamCity, Bamboo or CruiseControl.
Create build scripts for your projects (MSBuild, Nant, Rake).
The trial and error over-and-over of getting the final deployment story to work end-to-end.

Until now.

Welcome to OnCheckin.com

OnCheckin is a cloud powered Continuous Deployment service. Plug in your source control and webserver just the way you use them now, and let the service take care of the rest.

I created OnCheckin.com with the aim of bringing all of the power, reliability and repeatability of a build and deployment server to you and your team without any of the setup time, ongoing management, or cost. The operating cost in both man hours, and capital expense of outlaying for software and hardware.

I want to make it so easy to use Continuous Deployment for your next project, that you’d simply be silly not to jump on in.

What OnCheckin does

The gist of it is:

1. You setup a FREE account (no cost) that allows you to do continuous deployment for 1 website.

2. Give less than 5 minutes of your time to setup a project (depending on project configuration, less than 10 form fields!).

3. OnCheckin.com watches your source control for any changes, and if so it queues a deployment.

4. OnCheckin downloads your source, builds it, runs your unit tests, takes your site offline with a holding page that you configure and then deploys to your servers before starting up your website again.

5. Afterwards OnCheckin allows you to easily review build logs and unit test results online.

And I’d like to think the awesomeness from OnCheckin is just getting started - I’ve also got a bunch of new features on the way and I want you to have a say, so get in and let me know how you’d like the service to grow by voting on the Trello board.

And it works for you right now

Continuous Deployment shouldn’t be limited to enterprise or open source developers alike.

When building OnCheckin I’ve tried hard to make the service agnostic to developers of all kinds. Whether you like using Github or TFS, MSTest or nUnit, FTP or WebDeploy.

With this comes the need to have OnCheckin.com work for teams using all types of tooling:

Source Control Providers:

Github.com
Bitbucket.org
Privately hosted Git
SVN
Team Foundation Server

(Mercurial hopefully coming soon!)

Unit Test Frameworks:

MSTest
nUnit
xUnit
MbUnit

Ways to Deploy:

FTP
FTPS
SFTP
WebDeploy

Where OnCheckin fits in

Having spent many years working in software shops, web development houses, digital and advertising agencies, I know that it isn’t uncommon when working for a client to be forced to deliver great work while low on budget, short on resources, and often pushed for time.

Regardless of your situation, the mantra still sticks:

Deploy Early and Deploy Often

I created OnCheckin to help people building ASP.Net websites all over the world by giving them all the power of Continuous Deployment, without all the hard work or learning curve.

The cost of an account starts at $0.00. The time it takes to get setup and give it a try should be just minutes. I’ve worked hard to try and simplify the setup. If you don’t use continuous integration and deployment yet, I hope you’ll want to try OnCheckin for your next project.

It’s early days though – OnCheckin is in Beta

While the service has handled thousands of builds already (this blog itself is deployed using OnCheckin) and the service is setup to scale, mass varied usage is still going to be a relatively new thing for the service. As with most services like this that are just launching, I don’t expect it to be entirely kink free.

A ton of user testing has taught me that every person is a unique, special kind of annoying, unpredictable snowflake.
- Maggie Utgoff

There may be some projects types that won’t build yet. Any project type out of the box in VS2008-2012 should build fine though (see F# below thought) – outside of this, your mileage may vary.

There may be some types of source control repository products it may not fully support yet – there are a lots of different SVN servers, Git servers and TFS configuration's out in the wild, and I definitely don’t profess to have had access to them all during testing.

Third party project types won’t work – neither will F# projects just yet. If demand is high it’s definitely an option in the future though.

If you experience issues with anything, I’d love to help you get OnCheckin working for your setup – simply reach out by email (doug[at]oncheckin.com) or twitter @OnCheckinApp.

Technology stack

For those of you interested in knowing what's behind the scenes, I can happily share that I built OnCheckin using the following technology stack:

.Net 4.5
Entity Framework 5 Code First Migrations.
ASP.Net MVC 4
DotNetOAuth
SQL Server 2012
Windows Server 2012
MSBuild
SignalR
TopShelf
WebDeploy
BouncyCastle.Crypto
ELMAHR
Munq IOC Container (fast)
Moq
xUnit
Azure Service Bus

Serving duties are spread across a number of cloud based servers with SQL, Web, Build and Source Control spread into separate roles.

Head on over and signup for a free account

Special Thanks

I can’t take all of the credit for what I believe to be the awesomeness that is OnCheckin.com.

Over the past 6 months while I've been putting the service together, I've had to moral support of many great people. Some have given me feedback, some helped steer me on the right path. Others have just been great support whether you’ve actively known about it or not. Thank you to everyone who offered words of wisdom, or just shared a beer with me for support over the past few months, there are too many to mention but you know who you are.

Some special mentions I will make though (you may notice a trend):

Sayed I Hashimi
@sayedihashimi
http://sedodream.com

I cannot say enough about the awesomeness that Sayed gives to the ASP.Net community – he literally wrote the book on MSBuild. While working by day on the Visual Studio web team, he also puts in a huge amount of effort working with the community. Everywhere I’ve visited on the web relating to Visual Studio Build, MSBuild, TFS, you name it he’s written a blog post about it, answered a StackOverflow question about it, or replied by email or Twitter. Sayed personally took the time, more than a few times, to help me by replying to emails, Stack Overflow questions and tweets. Thanks Sayed!

Paul Stovell
@paulstovell
http://paulstovell.com

Paul needs no introduction, having brought Octopus Deploy into the world. Paul also took the time to sign up, take a look at OnCheckin and give me his thoughts. The input I really valued from Paul wasn’t just his feedback though – his “from the road” start-up advice and words of support were really what helped me keep putting in the hard yards. All of the months of less blogging, more coding, and obsessively long hours wear on you. Pauls advice motivated me to push on. Thanks Paul!

Daniel Nolan
@dan3r3
http://ready-roll.com/

Dan is probably another name you’re familiar with. Dan brought Ready-roll into the world and with it the awesomeness of automated SQL versioning and source control support. Dan offered some great advice, took the time to review the service and generally put my mind at ease about what I might be doing wrong, right or what was just plain crazy. I recently found out he also lives two blocks from my home, so me shouting him the odd pint or two should definitely allow me to tease out more golden nuggets. Thanks Dan!

Diary of a Ninja

Show Your Recipients You Care – Avoid noreply@

As old as time Itself

Why this is bad #1 – Emotional

Why this is bad #2 – The Dreaded Spam Filter

What we can do

Summary

Dear Microsoft, Please Include Web Deployment Projects in Visual Studio 2012

Visual Studio 2012 Web Deployment Projects – An Unknown

Have your say!

Autoplay, a New Type of Popup Plaguing the Web

“Crap!… mute, mute mute!”

People who do this stuff for a living – the W3C

The future of media on the web – HTML5 Audio/Video

A word on walled Gardens

Advice if you have to “go there”

Stop! Thief!

Micro Benchmarking Your ASP.Net Pages Using Apache Bench

Tooling tooling everywhere…

Info for nothing, and the tests for free…

Performance testing - Thinking it through

Making it all happen

Taking it one step further

Summary

Up Log Creek Without a Paddle – Part 1: Windows Audit Logs

Searching through Windows Auditing Logs

Finding a Paddle

2012 – Log Parsing; Now With GUI!

Example queries for Windows Security Logs

Next up -> Scanning IIS Logs to investigate security incidents

Further Reading

Up Log Creek Without a Paddle – Part 2: IIS Log File Investigations

Information overload…

Fields of Gold

Log Parser to the Rescue

First look: Security queries

Non security related queries

Summary

Visual Studio 2012 Web Deployment Projects are Dead – Long Live Publishing Profiles

Blinded by the light…

It’s time to move on… Publishing Profiles are actually better

Awesome things that get to join the party

The future looks bright

The Senior Position Fallacy

Microsoft TechEd Australia 2012

Swag time

On with the show

Setting up a VPN server on Windows 7 or Windows 8 – Secure your Internet use while away

How is this possible?

What you’ll need

On with it then…

You have been warned

Problems with .Net 2.0 Mixed Mode Assemblies inside Visual Studio .Net 4.5 Test Projects

The Problem

The Solution

Integration Testing FTP Connections in .Net

The Challenge

Run local. Run often.

Steps to make it happen

The Pitfalls of Cut and Paste Coding

Time to go on Safari

Following the paper trail

A copy of a copy, of a copy, of a copy

Teaching Your Application to Read

Let me get my glasses

Standing on the Shoulders of Giants

Input/Output

198 languages!!!!

Included Tooling

Summary

Reaching The Internet Event Horizon

Welcome to the Future

Entity Framework Code First Migrations in Teams

Implementation Details – Removed or Replaced?

What it’s likes when working alone

Code First Migrations in a Team

Action Plan A - Automatic Migrations

Action Plan B - Manual Migrations

Summary

SimpleDeploy.Net - An FTP Style Wrapper for the MS Deploy API