< Part One: Medium Trust primer

< Part Two: Devising a solution

Welcome to part three! We’re going to get some work done on unit testing in Medium Trust. If you want a primer, check out the previous posts. If you came here from there, well hello there recursive sentence!

First things first: the code for the below is all in MIT-licensed Umbraco 5! As you may / may not know, Umbraco 5 will be shipping both a set of Framework assemblies in addition to the core CMS when we release later this year. You can use it all for free, isn’t that grand?

For today, here’s a big unmissable link to download a preview zip of Umbraco.Framework.Testing. It’s MIT licensed and you can use this in your tests real easy. I’ve put it on my DropBox for now – I want to get this post out quick and go to the pub! into your hands! In an upcoming release we’ll make Framework & CMS downloads available as individual zips on CodePlex.

A quick note: I only came up with this last week, so (ironically) Umbraco 5 does not yet support Medium Trust as of today’s Alpha 3 – but the point is, we’ll now be able to use this technique to test our code in partial trust and make it happen reliably, and soon.

If you’d like to download all of the Umbraco codebase and see what other goodies are in our Framework and CMS, visit the repository at http://umbraco.codeplex.com/SourceControl/list/changesets.

To use this in your NUnit tests

I’ll walk through the code in another post, but if you want to jump straight into using this, download the zip, add a reference to Umbraco.Framework and Umbraco.Framework.Testing and you’re set.

Note: Because your tests will call NUnit’s Assert methods, NUnit itself needs to be able to run in partial trust. The zip contains a rebuild of NUnit.Framework.dll – the only change from 2.5.10.11092 is adding the AllowPartiallTrustedCallersAttribute to their assembly. Don’t believe me? Check out my fork at https://bitbucket.org/boxbinary/nunit-2.5x

Your test runner does not need to be rebuilt, so this is still compatible with NUnit’s runner, ReSharper’s runner, TeamCity etc.; only your own code referencing the NUnit Framework needs to reference the assembly.

Once you’ve added the references, let’s make a new fixture called MyCoolFixture. Set the fixture so it inherits from AbstractPartialTrustFixture, like so:

Those TestSetup / TestTearDowns are there for you to put relevant code or just return if nothing’s needed.

Next let’s add a test or two:

Nothing crazy, just a simple thing which should work and a simple thing which should not.

What, pray tell, should not? Well, that there Clone method is a really handy, and extremely common extension method that uses binary serialization to give you an exact copy of an object. FluentNHibernate uses it, in fact. But BinaryFormatter tries to set private members, and as we now know you can’t do that in Medium Trust!

So in theory that second test should fail. Let’s run that to see if all my promises are true:

Check .. that .. OUT!

SetUp : Umbraco.Framework.Testing.PartialTrust.PartialTrustTestException : Test ThisShouldNotWork fails in a partial trust environment, due to: Request for the permission of type ‘System.Security.Permissions.SecurityPermission, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089′ failed.
—-> System.Security.SecurityException : Request for the permission of type ‘System.Security.Permissions.SecurityPermission, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089′ failed.

So let’s fix the test so TestShouldFail() appropriately asserts that … it should fail:

And run it:

Woo!

But what if I want some tests to literally only test something in full-trust?

Well, you can of course have full-trust tests in a separate fixture. OR, you could decorate the test with the [TestOnlyInFullTrust] attribute I’ve added to our library. I’ll change the test again to demonstrate:

And run:

Sweet!

So, there you have it. Easy partial-trust testing by inheriting from one base class.

In the next post – which may be next week, at this rate – I’ll walk through the code and explain how it works, for example if you’d like not to inherit from another class and just want to spin up an individual test yourself in a partial trust context individually.

We can also look into making this work with other testing frameworks, if you’re really nice.

Hope this helps you; let me know if so!

< Part One: Medium Trust primer

> Part Three: Download the code

Welcome to Part Two, wherein we will tackle the beast that is Medium Trust in our NUnit tests. If you just want a downloadable zip to get going, you can always jump to part three.

Based on what we learnt in the previous post around how the permissions are stored in configuration, a simple option is open to us. The basic premise is that in our unit test, we’re going to spin up an AppDomain and pass in the same permissions from configuration that the ASP.Net hosting engine uses. We’re then going to run the test method in that partially-trusted AppDomain, rather than in the full-trust AppDomain of the test runner.

Simples! The end. Bai!

Well, not quite. First, some gotchas.

Spinning up another AppDomain is called “sandboxing” for a reason: they are totally separate, and any communication between AppDomains has to be done via .NET Remoting using serializable objects, just like using your own Remoting channel, WCF or a good ol’ SOAP web service on another machine. Can you spot the first gotcha yet?

Gotcha 1: You can’t just use AppDomain.SetData to pass objects by value from your test to a Medium-Trust domain because it doesn’t have permissions to deserialize them fully.

Makes sense when I put it like that, but it took me an hour of wasted time before I facepalmed and almost quit coding forever.

There aren’t ways around this. Passing an object to another AppDomain, whether it’s via the SetData and GetData methods, requires the object to either be serializable (if you want to pass it by value) or inherit from MarshalByRefObject so that you can deal with the object as a regular Remoting proxy to the other AppDomain.

Part of the point of spinning up the other AppDomain is precisely to prevent certain code from running, you know, like serialization that can get/set private members.

Yeah, I wasn’t proud. A whole hour.

So the next idea was to use the power of MarshalByRefObject to instantiate something in the partially-trusted AppDomain, and control it from the original fully-trusted AppDomain of the test runner. This is pretty much how we’ll solve the problem, but wait:

Gotcha 2: It’s tempting, but we can’t just make our entire test fixture inherit from MarshalByRefObject.

Let’s agree that it would be a complete pain in the arse to have to replicate the nice, established features of a test runner. Agreed? OK, with that sorted: we can’t have our test fixture inherit from MarshalByRefObject to solve this problem.

We need the partially-trusted AppDomain to “own” the instance of the test, and run it there. If we’re going to try to use an existing test runner (NUnit’s, ReSharper’s, TeamCity’s, etc.) then our first point of entry into a test is when the fixture is already running.

The solution workflow

The concept I’ve arrived at is, as always once you’ve spent ages banging your head against a wall, quite simple in hindsight.

Here’s how the workflow will happen – again, this is just within a standard test runner:

• During fixture setup:
  • Our fixture will get set up by running code marked with the [FixtureSetUp] attribute
  • We’ll then create the partially-trusted AppDomain
  • We’ll tell that AppDomain to create a new instance of a simple marshalling class that can run a method for us when given a MethodInfo class. By calling AppDomain.CreateInstanceAndUnwrap(..), we’ll get back a Remoting proxy to that object so that we can control it from our original test runner.
  • These will both live as fields on the test fixture that we can access from each test.
• Before each test:
  • Out test setup code, still running in the full-trust AppDomain, will get run because it’s marked with the NUnit [SetUp] attribute
  • At this point, we’ll get the name of the current test from NUnit’s TestContext, and tell our simple marshalling class to do two things over in that partial trust AppDomain:
    • Run a TestSetup method, to make sure if it has “real” setup work to do, that’s still fine
    • Run the actual method. We get that by using Reflection in our full-trust AppDomain to grab the MethodInfo of the current test name from NUnit’s context.
  • Our marshaller will then run the exact test method in the partial trust domain.
  • If the test fails, we can figure out why, and if it’s to do with permissions, BAM our time has been officially saved

“But wait!” I hear someone wail at the back, “By running the test in the SetUp part, you’re now going to run the test again in Full Trust!”

Well, I thought of that.

NUnit has a funny thing called a SuccessException. It’s how it guarantees that if you call Assert.Pass() in your test, the rest of the test will likely not execute – pretty much what you’d expect.

So, in our “management” set-up code, we’re going to intentionally throw a SuccessException. I took a look at the NUnit codebase to clarify, and sure enough that prevents the test from executing a second time.

The code, where’s the code?

Well, I thought I would just give you theory. Kidding! I’m hilarious. The downloadable code is on the next post – I really need to make my blog more developer-friendly, so for now it’s a fresh new page for us.

This post originally started out life as a three-paragraph thing, but then I realised the topic really merits a lot of background, so I thought I’d get with the cool gang and do a multi-part post.

First, I’m going to cover a bit of background about Medium Trust, and how to understand what it actually means for your code in terms of permissions.

The second post covers a solution I’ve produced over the past week and in the latest build of Umbraco.Framework.Testing which allows you to easily ensure all your NUnit tests run in partial trust.

The third post has the download and some instructions, so jump there if you want to cut to the chase.

In these posts I’ll be talking about .NET 3.5 and above – just to be clear from the outset.

Medium Nightmare

Medium Trust is probably one of the most infamous parts of an ASP.NET developer’s coding life. If you work on your own applications, hosted in Full Trust environments under the control of you or your team, you’re fine. You may have never even come across an issue. It’s this phenomenon which has created such a painful situation: so many developers write code which they reasonably expect can be deployed anywhere; it passes its tests, and works on their machine.

But if you’ve ever put together a website with some cool fandangled bit of code that runs fine locally, and then you’ve deployed it to your hosting account, only to find your beautiful creation misses you dearly and is just sat there screaming complaints of SecurityPermission this, and FieldAccessPermission that, you’ll know how tough it can be.

For many it’s the beginning of a very long and arduous process of re-engineering and debugging in what can seem like an infinite loop, and many assumptions about the way the application works (such as lazy-loading proxies in NHibernate) often need to be entirely revisited. Or not, depending on how much luck you have with your Bing search. It’s another large part of the problem: it’s such a minefield, even searching for solutions can be spotty.

It’s not all in your hands either; it’s an easy mistake to make precisely because it’s so difficult to test in a reliable way and it’s so easy to begin developing in the default setting for many ASP.Net project templates, which is: Full Trust. Fluent NHibernate, AutoMapper, certain setups of Castle DynamicProxy, of NHibernate – these are Big Name Libraries, and they all have problems. We’ve all been there: Umbraco included up until the past year or so. It’s a long list – and unpicking the problems one by one, rebuilding, and pressing F5 in your browser is a pain in the arse.

I don’t like pains in my arse, and since we’re busy building Umbraco 5 right now, I’d rather we don’t become one in yours either. I figured the other features we’re coding have test coverage, let’s get the Medium Trust issue covered too. And since it’s all open source, everyone can do it. Yeay!

Security background – what exactly is Medium Trust and its permissions?

“Medium Trust” is a loose term for a set of Code Access Security policy settings that offer less freedom to bend the server to your every will. It’s complicated by the fact that what constitutes “medium trust” is actually a set of permissions listed in configuration, and some hosting companies tailor the settings, so you don’t quite know what you’re dealing with. But in general the policy means at least the following permissions:

• AspNetHostingPermission of “medium” – For example, a minimum requirement that calls to public types in the System.Web namespace must be coming from code that is granted at least a certain “minimal” trust level.
• A DnsPermission which is unrestricted. Go mad on those Dns lookups.
• An EnvironmentPermission which grants access only to the following system environment variables by default:
  • TEMP; TMP; USERNAME; OS; COMPUTERNAME
• A FileIOPermission that says you may only Read, Write, Append and browse the filesystem that starts with the root of your website and nowhere else. Shocker.
• An IsolatedStorageFilePermission which permits isolation of storage first by user, and then by code assembly. It allows you a data store for the assembly that is accessible in any domain context.
• Bizarrely, a PrintingPermission which permits programmatic printing to the default printer.
• A SecurityPermission which allows the following flags:
  • Execution (to allow managed code to execute – yeay!)
  • ControlPrincipal (to manipulate the Principal object i.e. set the current logged-on user)
  • ControlThread (to allow the totally unambiguous “certain advanced operations on threads”, thanks MSDN!)
  • RemotingConfiguration (to configure Remoting types and channels)
• An SmtpPermission which allows you to connect only to port 25
• An unrestricted SqlClientPermission
• An unrestricted TypeDescriptorPermission
• An unrestricted WebPermission (confused? This actually signifies unrestricted access to make HTTP requests to other services)
• And finally the one which crops up most often, a ReflectionPermission with flags of RestrictedMemberAccess. This signifies “partially trusted code can access nonpublic types and members, but only if the grant set of the partially trusted code includes all permissions in the grant set of the assembly that contains the nonpublic types and members being accessed”. Again, thanks MSDN! In practice, this means you can’t access private members of someone else’s assemblies – and this is what brings down most libraries.

What’s important about this set of permissions is actually what it doesn’t give you. For example, let’s look at some of the SecurityPermissionFlags you could have won:

• Calling unmanaged code
• Ability to provide serialization services
• Ability to create and manipulate AppDomains

Or the “MemberAccess” ReflectionPermissionFlag, which would have allowed you to use reflection to operate on all members of a class (even its privates).

Also, no OleDbClientPermission or OdbcClientPermissions are in there by default – most hosting companies add these to their Medium Trust configurations.

And certainly no RegistryPermission.

How do you know this and where is it configured?

Well, perhaps the wider point is I don’t know this by reading one handy straightforward piece of simple documentation. The top Google result for me is a piece of “retired” MSDN content left there for reference or for anyone who has run out of everything else to read in the world ever. The top Bing result is an MSDN video from 2005. I’m sure it’s very good, but it’s an exe file download. From 2005.

90% of what you need to know about Medium Trust’s permission set is stored in the web_mediumtrust.config file for the .NET version on which your application is running. On my machine, for .NET 4, that’s C:\Windows\Microsoft.NET\Framework\v4.0.30319\Config\web_mediumtrust.config.

Web_MediumTrust.config explained

The config file has a simple job: declare named sets of permissions. It does this by first declaring a list of SecurityClass elements to predefine the type names of the permissions and give them a key, and then goes on to declare PermissionSet elements which refer back to those types and pass in arguments. Here is the one installed on my dev machine in full:

When ASP.Net spins up the AppDomain for your website, it checks the <trust> element of your web.config file (which may be inherited from a parent config that the hoster has set), and then goes to the appropriate config file to load the right set of permissions needed for the AppDomain. After that, it’s all down to security which is built-in to the CLR which transparently checks the caller’s permissions vs the callee’s permission demands when running managed code.

Can you see where I’m going with this?

Let’s get on to how and why this matters to us. In the next post I’ll outline how we can use this information to restrict tests to a very similar set of permissions.

Wow, what a rollercoaster so far. I thought I’d put up a quick update on here, as although Fed has been updating our team blog, I have limited Azure / Twitter / programming access (more on that in a minute) and I know this will go out via Feedburner automatically to anyone who is interested in our progress, so given the current set of news we have to share, this is the place to let you know. Luckily I managed to remember the password!

So, first of all it’s day 18 and we’re still not in Russia. This was not really part of the plan; we should have been in Russia a while ago in order to be “on track”. But things got hairy pretty soon after we departed England. Our convoy of six ambulances had headed south-east through Europe in order to get to Budapest, at the time a relatively small diversion in order to meet with the GoHelp charity and other teams outside of our convoy, for an annual event to share a few drinks with everyone who embarks on this mission.

Budapest is a lovely, lovely place, but unfortunately as Fed wrote at the time, it stole more than just our hearts: someone forced the back doors of our ambulance overnight and stole my suitcase, my hidden rucksack containing my laptop which I’d intended to use for blogging (and some Umbraco 5 work) and our camera chargers, Fed’s Kindle, and perhaps the most devastating thing given the context: our vehicle documentation.

Given that all these things except perhaps the suitcase had been hidden, we didn’t actually discover the theft until we were about 300 miles away from Budapest near the Ukranian border. At the time, it was a whirlwind of stress. Any theft is always earth-shattering but the full extent didn’t actually dawn on us until we had attempted a few border crossings: that vehicle documentation, a “V5C” in the UK or “auto-passport” in local parlance, was the key to us leaving the EU.

As it happened, the entire convoy spent about 2 days bouncing between the EU and Ukranian border in no-man’s land because the remainder of the convoy had “temporary V5″ documents whilst our permanent ones were being mailed to catch us up in Volgograd, and we had been given advice that this (and our hastily-obtained printout of a V5C scan we had on file) would not be a problem. But with clarity of hindsight we now know that actually the Ukraine was never going to be easy: it levies a 3000EUR fee per vehicle for crossing with temporary documents. Well, I say a “fee”: they claim we would have received a refund when getting to Russia in exchange for proving that we weren’t going to dump the vehicles in the Ukraine, but we just didn’t have that kind of cash or trust in chance to continue. Plus, it was probably a lucky break: we now know that our single ambulance would never have made it from the Ukraine into Russia without the original V5C and Fed and I would have been stuck having to turn back.

So the whole convoy chose a new route, after some deliberation between going north or south of the Ukraine to a Turkish ferry. We decided to travel north through Slovakia, Poland, Lithuania and finally into Latvia where there are several border crossings with Russia that avoid Belarus. Here the other five ambulances successfully made it through after a painful 9-hour queue, but Fed and I in ambulance #5 were turned away. It was an incredibly gut-wrenching moment, and everyone was incredibly supportive – although clearly we were all upset. Since that time, Fed and I have been traveling on our own with determination, to a few of the other Latvian crossings. At one border we even made it to the very last step (luckily after only a 30-minute queue), but still got turned away.

Since Saturday when the others made it through the Russian border and beyond we’ve tried all sorts, but finally on Monday my mum and dad managed to make a breakthrough with the DVLA in the UK who kindly agreed to get a new V5C sent to our London home first-class, where Fed’s mum and sister have been staying. Thanks to them all it’s now with DHL and we expect to receive it before 12pm tomorrow.

Through all of this we’ve had our ups and downs. Of course, the original robbery has always been at the back of our minds, but perhaps if we’d had the permanent V5C on us it might have been that which was stolen. But there have been other things, the largest of which is probably that our Russian visas expire on the 12th of August. Looking at the mileage, it’s still do-able if we enter Russia tomorrow night, but there’s no doubt that it will be a case of doing a lot of driving in a short amount of time. Which is why it’s so incredibly kind that one ambulance has stayed to wait for us in Samara. One of their team, Scott, will fly back to Moscow in order to jump in our van and help with the driving to get to Samara as quick as possible, and everything else being well (we are definitely due a lucky break!) we will then continue as a convoy of two vehicles and six people – a much more reassuring prospect. Together we probably won’t catch the other four, but the idea that it’s still at least possible to get all six vehicles there is a glimmer of hope to which we are all clinging right now.

It’s been our intense desire throughout all of this that we make sure that we get this vehicle to Ulaanbaatar, to deliver it but also to visit the places where the donations to our and the other teams’ fundraising websites will be spent. We’ve still got other backup options in case DHL lets us down, for example donating the vehicle to a needy beneficiary in the EU and then traveling to Mongolia via train or plane, but of course none of them quite lives up to the visions we’ve had for the past 18 months. It’s about the journey as well as the destination, and – whilst there are still hurdles we must cross – potentially Team 2 and our parents back in the UK scrabbling to help us out will have made this current hurdle conquerable.

And with that I’ll bring this post to a close. Since I’ve been doing all of the driving, and had my laptop stolen, I’ve not been able to blog or tweet at all. But hopefully the next time I get chance, it’ll be with good news. Fed has also been doing sterling work taking lots of great pictures for our team blog which is here, which she updates from the passenger seat whenever there is a sniff of open Wifi.

Until the next update, all the best and thanks again to everyone who has donated funds, to our amazing convoy-mates and our parents for their help, to all those kind folks we’ve met along the way, and to my incredibly supportive employer Umbraco to make this mission so worthwhile.

This year my girlfriend @federicacocco and I are driving the ambulance pictured below from London to Mongolia for charity, in order to donate the vehicle and any funds we raise to a community in need.

Please spare a few £ / $ / € and help, it’s really easy at our Team page: justgiving.com/muttley-to-mongolia

Below is the ambulance we’ll be donating, setting off on July 9th 2011 and arriving roughly 4-5 weeks later. We’re doing this in partnership with the London Ambulance Service and the registered UK charity Go Help.

Thanks to everyone who has donated so far, including a very generous £250 from my awesome employer, Umbraco. We appreciate all donations large or small, so please do give something today – thank you!

Our route is below. Not long, eh? We’ll also be tweeting, blogging and filming along the route. If you’d like your company to sponsor us, we can accept wire transfers or PayPal Donate transfers too – please do get in touch as we’ll pretty much bend over backwards to accommodate your sponsorship request. There’s a lot of space on the side of that ambulance for logos etc.!

We’ll be liveblogging the journey at this tumblr: http://mongoliarally.tumblr.com/

A little while before the fantastic CodeGarden11, whilst prepping the first Umbraco 5 CTP with the team, I had the humbling surprise to be nominated to become a member of the Microsoft Regional Director programme.

This was a fantastic compliment and one which definitely came out of left-field. In the coming months I’ll be increasing my blogging output and applying for as many speaker slots as I can at the many fun tech meetups with which our industry is blessed – including spreading the word about Umbraco which is, of course, one of the most popular CMS tools on the ASP.NET stack. *Cough* bias *cough*.

A little about the programme: first of all, I don’t work for Microsoft! All RD’s have jobs at, or own, technology-focussed companies using Microsoft tools. I love my job at Umbraco guiding the architecture of version 5, and feel like the luckiest software guy out there to be spending my days working with such a great team and brilliant community.

The programme is similar to Microsoft’s MVP scheme, with the key difference being that RDs are a smaller group (around 140) and tend to chat with people who are not yet users, and are evaluating the switch to Microsoft technologies, or software based on their stack such as Umbraco.

There’s a large crossover though: both programmes are about evangelising the areas that Microsoft does well, and acting as a conduit for feedback to the MS product teams. Although it’s tempting to believe otherwise, we of course have the freedom to speak out where we believe Microsoft has made mistakes or can do a better job.

I have to say, it’s humbling to be a invited to be amongst such prestigious company. I’m going to have my work cut out keeping pace with their bright minds and speaking talent, but I’m relishing the personal challenge.

As a final side-note, it was really nice to see that within the first 24h of “meeting” everyone on the mailing list, several RD’s replied back saying how they use, and love using, Umbraco. Smiles all round!

This year’s Umbraco festival, CodeGarden11, was another fantastic success and as ever, brilliant fun. Amongst all of the great sessions and the happiest tech community on the planet, it’s difficult to take just a few highlights, but here’s a couple of things which stood out for me:

The Retreat

Each year the Umbraco HQ flies out the Umbraco MVPs who are nominated by the community, together with other active contributors, for a few days of drinking, BBQing and hacking in Denmark prior to the CodeGarden festival. It’s one of our ways of saying “thanks”, and it’s always great to get everyone together from all over the world. We also have a tradition of playing a bit of football, so this year we went the whole hog and got shirts personalised for everyone. I’m the one second from right, sucking in my fat takeaway-induced belly.

We played shortly after this photo was taken. I wish we had a proper “before” and “after” shot, since half of us flagged during the game and the rest were shadows of their former selves, but we did get another kind of team photo done at least:

This was one of the bingo prizes

The keynote announcements

The video for the keynote is up and has some placeholders for you to jump straight to each section.

Courier 2.0

Per from the HQ has been hard at work for many months (in addition to welcoming a new baby girl into the family – the guy is quite prolific) and the resulting Courier 2 deployment tool, released shortly before CodeGarden itself, has already been a runaway success. Check out the details page here, and the keynote video above.

Projects 2.0: The Umbraco Deli

Likewise Peter and Paul from the HQ have also been hard at work (although they are lagging behind Per as no new children this year yet). At CodeGarden they launched the new marketplace for commercial Umbraco packages, codenamed the Umbraco Deli. It’s a really straightforward and easy-to-use tool if you want to have your Umbraco plugin sold through a central repository. Umbraco HQ takes a small percentage of proceeds and provides you with all the licensing and admin tools you need, together with great stats.

Umbraco 5 CTP

Finally, not wanting to be out-done on the hard work front, myself, Shan and the rest of the team have been busy building version 5 of Umbraco. We released the first Community Technology Preview during the CodeGarden keynote, and the feedback so far has been humbling!

You can find session videos on the CodeGarden11 site for all of the above, including the great sessions by our guest speakers too – and we’re adding more each day.

The boat-party

Ah, the boat party. Always fun to grab a load of drinks and relax after the first day of CodeGarden whilst having a tour around the wonderful city of Copenhagen. This year unofficially sponsored by Tuborg and Lonely Island:

The boat-party, part 2

I took this picture at 6am after the last day after an awesome night out

At the precise time of writing (6.45pm GMT) the cablegate.wikileaks.org site is firmly unresponsive from its recent Amazon home, and a global DNS check shows that changes to its domain are currently propagating round the world – back to Bahnhof (RIPE check here).

Bahnhof is the Swedish ISP originally heralded in the media as WikiLeaks’ chosen hosting partner some time ago, thanks in part to its futuristic press shots of nuclear bunkers and movie-set lighting.

MSNBC is speculating that it might be because Amazon has taken the decision to force the site off its services. Amazon has certainly had its fair share of press coverage this time round – definitely more than the last time it happened with the Warlogs. My blog alone has received quotes and trackbacks from the Wall Street Journal (twice), The Guardian, The Register, The Seattle Weekly, and Fox News – and there are countless other outlets covering the issue too.

Whilst it’s not unusual for WikiLeaks to keep moving their services, this is the first time their use of Amazon’s servers have had such a prolonged outage despite WikiLeaks’ own claims of a huge DDoS attack. Moving to another host directly as a result of downtime would suggest unresolvable problems at Amazon.

Considering no reports of outages at other Amazon-hosted sites are proliferating, you’d be forgiven for suspecting that this time it might just be personal.

Update: If this tweet from @SkyNewsBreak is to be believed, the US Dept of Homeland Security has confirmed that Amazon has agreed to stop hosting the content. The nature of the downtime would suggest that very little notice was given before the action was taken.

Update 2: WikiLeaks themselves just tweeted effective confirmation. Does this bring the saga to an end…?

A month ago WikiLeaks added to their burgeoning notoriety by launching the controversial Iraq war logs. Completely by accident I noticed they were hosted on Amazon servers, in their Ireland and US datacentres, and blogged about it here.

The story got picked up by The Register, The Telegraph and then spread to various other places. Even Paul Mutton, the guy who blogs for NetCraft, picked up the trail and has been blogging about it since on the NetCraft.com homepage.

I thought it was a Big Deal – to me it seemed so odd, surely a mistake, to put this material not only on servers run by a US company, but physically on US soil – surely making it quite difficult to refute claims of illegality by the US authorities.

And then, tonight, they go and launch the ‘cablegate’ US Embassy logs – and it appears that to WikiLeaks using a US hosting company like Amazon isn’t a source of worry at all.

The new site, cablegate.wikileaks.org, is hosted on the same Octopuce and Amazon US IP addresses I reported last month. The wikileaks.org homepage is currently spread across two Amazon Ireland IP addresses.

I had thought that the slow removal of the Amazon servers over the past few weeks was demonstration of Amazon’s quiet refusal to host the material; I was evidently wrong. So much for the “Swedish bunkers”.

I’m not going to prattle on much about WikiLeaks, but I noticed something again about the recent WikiLeaks US Embassy release.

Quick disclaimer: it’s just harmless gossip, I’ve nothing to do with WikiLeaks, and may even be wrong.

So I was scanning through one of the cables on The Guardian’s coverage of WikiLeaks’ latest release, and noticed a hyperlink at the bottom of the cable.

This hyperlink refers to the “SGOV.GOV” SLD which refers to SIPRNET, which apparently is the internal network used for diplomatic purposes.

I’m not an expert, but that looks like a MediaWiki address (of Wikipedia fame) to me.

I just kicked off some blogging about Umbraco 5 over on the official Umbraco blog.

First up I’ve done an introduction setting the scene for those who couldn’t attend the festivals this year, and then I’ve posted a Q&A for November (hey, there are still a few days left…)

Next week I’ll be posting some technical details on our data modelling.

In my post last Friday I detailed how I’d noticed that WikiLeaks was using servers at Amazon in the US and Ireland for its latest release. I noticed completely by chance and had the good fortune to post it within the hour of their launch, and since then a few different sites have picked up on the story.

Netcraft since blogged about the find here, as did The Register, and most recently The Daily Telegraph. I also managed to get a question asked via Twitter during the Frontline Club press conference with Julian Assange but unfortunately some confusion led to a mixed answer.

With the increased press attention to the matter I’ve kept an eye on the domains every once in a while. As of today, the Amazon US mirror at 204.236.131.131 just got removed from one of the sites http://warlogs.wikileaks.org leaving only Ireland and France. The Irish IP is still run by Amazon, a US company.

Right now, the Amazon US mirror is still on the main site at http://wikileaks.org, but I’ll update if this changes.

5:05pm BST

5:20pm BST

5:20pm BST

[Update 7 - WikiLeaks have removed one of the US mirrors shortly after a Telegraph story went live - see my latest post, you might like to read the below first for context tough]

It’s been a tough week. My work backlog has grown since I’ve been ill most of the time, and I’ve been trying to catch up. However, just as I was going to bed tonight, my girlfriend was still at her office in Paris. She’s been writing some press coverage of the latest WikiLeaks release, and since it all went live tonight, I took a break to check out some of the work.

Techie as I am, eventually I branched off from looking at the press stories to see where the WikiLeaks Warlogs site servers were hosted – and since I was expecting it to be Sweden, I was somewhat blown away. It’s made me question if I’m still full of Ibuprofen, but I had to put this up to get some feedback.

First of all I checked Netcraft. This reported it was hosted on Amazon EC2 in Ireland. Ireland? Amazon Web Services? I thought it was wrong.

So then I checked what IP addresses were being returned globally and this is where it got really strange.

Originally when I wrote this there were four mirrors, but now there are five. The weird bit? Three out of the five are with US-owned company Amazon’s EC2 hosting service. A US-owned company. Two of those – two out of the five current mirrors – are on US soil in their west-coast datacenter. The remaining two are with Octopuce in France.

Isn’t it strange that WikiLeaks would host its latest inflammatory content on US soil, or even that some of the mirror servers are owned by a US company? WikiLeaks has taken great pains to ensure hosting of content in Sweden or other “friendly” jurisdictions in the past, and in fact their donation link is reporting itself as in Iceland. The US and Ireland – and France for that matter – don’t seem like the safest “haven” for this data to remain online, after all in the past we’ve been told those “havens” are Iceland, or Sweden. What happened to those reputed Scandinavian bunkers?

To be clear, this set of IP addresses isn’t a CDN. WikiLeaks isn’t giving French IP addresses to French visitors to aid in speed; this is a round-robin DNS spreading traffic crudely across multiple hosts around the world to cope with traffic and DDOS. What’s weird is that none of the IPs are in Sweden, and all of them are in principalities with relatively straightforward cease-and-desist legislation, and for any of them to be in the US seems absurd.

To caveat this, these are their front-end IPs. These nodes could simply be serving data retrieved from those reported bunkers. But it still seems strange to me. Conspiracy theorists might ponder if it’s just asking for trouble. The US’s position, reported tonight by the BBC, is that “WikiLeaks has committed a crime by publishing stolen documents”. If true, why exacerbate it by doing so on US machines? Hasn’t that now just brought the issue smack bang into US jurisdiction?

Maybe it’s the Ibuprofen and I’m missing something – but this seems very odd to me.

Traceroute from Seattle terminating in AWS's west-cost region

[Update 1 - Saturday 23rd] - WikiLeaks.org itself has now switched to these same servers, meaning the primary public-facing URL is now in part hosted inside the US, and in part by a US company.

[Update 2 - Saturday 23rd] – Another mirror has been added in Amazon’s west-coast US region increasing the amount of traffic being served from inside the US.

[Update 3 - Saturday 23rd] – My post has been picked-up by cryptome.org on their homepage

[Update 4 - Monday 25th] – My post has been picked up and covered by The Register

[Update 5 - Tuesday 26th] – Netcraft has picked up my post and concurred with the location data

[Update 6 - Tuesday 26th] – Telegraph story links here too

[Update 7 - Tuesday 26th] – WikiLeaks have removed one of the US mirrors shortly after the above Telegraph story went live

I spoke last Friday at the UK Umbraco Festival, giving a snapshot of version 5 which is currently in development.

It was a great event and a testament to the powers of Warren Buckley (Xeed) and Adam Shallcross (The CogWorks) – their Umbraco community skills amaze me.

Here’s the synopsis of the talk below, here’s my slidedeck, and the video of the talk can be seen here at the Umbraco Stream site.

“Umbraco 5 is going to become the Umbraco you know and love on the outside, with a leaner body on the inside. In this reasonably techie talk I’ll go through some of the architecture choices, and I’m sure at one or two points there’ll be Visual Studio and a browser up on screen to show everyone progress so far. It’s early days so we’ll be looking at the concepts in progress, but Umbraco 5 will be pluggable to the nth degree, so this talk will give a bit of insight into where things are headed and how we’re going to get there. For anyone interested in how we maintain a global team I’ll also give a quick insight into how we organise the project and explain about some of the upcoming milestones for the rest of 2010.”

There were great talks including the keynote by Niels in which he shows the upcoming Windows Phone 7 app for Umbraco, and skinning coming in version 4.6. Many of the others are available at http://stream.umbraco.org – check it out. Thanks to all who came and made it a really interesting day out!

As of right now, Twitter in my industry sphere is going mad over not just the iPhone OS 4.0 annoucement, but as part of it Apple’s revision to their developer license agreement. Apple now insists that any applications allowed in the AppStore must henceforth be written in coding languages stipulated by Apple, and they cannot be produced in a third-party application which translates other code to those languages.

There are many points around whether or not this harms competition, but I don’t want to dwell on that here. My point is a little different: Apple just pulled the rug from under a valuable media partner in Condé Nast US. Why? Because Apple’s announcement not only scuppers the likes of MonoTouch, but it derails Adobe’s upcoming release of Flash Pro CS5 too, with its much-vaunted compile-to-iPhone functionality. It’s this media angle which I think is a particularly repulsive part of their move, because it’s not just ‘pesky Flash developers’ who want easier access to write apps for the iPhone and iPad. It’s publishers, too.

Nearly two months ago Condé Nast US went public with a (rather cringeworthy) video of their iPad plans. The article states “tablet”, but make no mistake – check the URL – this was iPad through and through. I should know; I used to work there. Despite its clear naivities in effective finger-based navigation as so deftly pointed out in the Engadget coverage, and whether or not you agree with the timing, Condé Nast announced that day not only a clear and sensible strategy but a key partnership in their adoption of Steve Jobs’ magic pill: joining with Adobe to write the application in AIR, meaning they could publish to the iPad as well as other formats – without wasting investment on two identical products for competing devices being made and filled with content every month.

The saviour of media?

With typical aplomb, Apple has pitched itself recently as the champion of publishers whose traditions are vested mainly in print. For a long time many of these publishers – typically now some of the most entrenched and embattled – have been bourgeois Gentleman’s clubs accustomed to having their pockets lined by the hard work of unpaid interns, with magazines flying off the shelf and a queue of fresh blood to keep the seats warm if any staff should decide they want a fair salary. But as has been documented well, best in fact by some of the nimbler online publicitions, that position has become threatened in the past five years. Headcounts have had to be frozen; great and under-paid journalists have been made redundant. Largely to protect the salaries of the upper management from dipping below £500k, sure, but it’s enough for them to have been chilled into some form of reaction.

Earlier this year Steve Jobs went on a crusade to win them over; a mission to convince them that he could show them how they could make up for time and all their missed opportunities. “Music labels love iTunes, I can do the same for you!” et cetera, et cetera. The likes of the NYT and Wall Street Journal, even the AP, have clearly seen promise and have welcomed their new partner. He’s convinced them that he has the iPad-shaped key to save their content business.

After all, content is king, no? Well as this deftly-hidden, unannounced aspect to their latest fanfare shows, actually it’s Jobs who is King. He may make a few publishers a bit of cash on the side, but in his personal crusade against a lumbering Adobe, he just cost one very important media partner a lot of money in wasted effort.

We can only hope that the attention drawn by the Emperor’s new clothes doesn’t convince some of our great institutions to make their swan song an iTunes exclusive.

Getting a development machine set up is a fine art of patience and a good memory. I personally recommend Ninite which is frankly awesome, followed by Visual Studio in its various flavours, followed by the Web Platform Installer.

However, with so many tools, frameworks, patches and add-ons to install it can be easy to miss something.

Even more frustrating, sometimes it can even be easy to install things in the wrong order.

Silverlight debugging: Get the right Tools

Specifically for Silverlight, it can appear as if you have all the necessary pre-requisites to create and debug Silverlight applications in Visual Studio 2008 SP1, but in fact your breakpoints are never hit.

You have the Silverlight project templates installed, you’re happily mixing and matching Visual Studio and Expression access to your project tree, but when compiling and running your project, Visual Studio either flatly ignores your breakpoints or reports that the symbols haven’t been loaded and that the breakpoints won’t be hit.

Some standard solutions

A lot of Googling takes you to answers around the following, none of which have actually applied to me recently:

• Ensure you are running from a http:// address rather than a file:// address
  • In summary, do this by creating a separate Web application in your solution and then adding a link from it to your Silverlight project to ensure VS launches your browser pointing at a web server rather than the filesystem.
  • Debugging is disabled by default when going to local filesystem-based URLs.
• Ensure “Silverlight” debugging option in your website project Properties > Web tab is checked
  • Some solutions even point to unchecking, then rechecking the option in order to kick VS into recognising the breakpoints.
• Manually attach to the browser process, since VS may have attached to the wrong instance.
  • This especially seems to hit IE8 users where Silverlight apps are reputedly launched in a separate process to the main HTML.
• Ensure you have “Silverlight” attachment type listed in the “Attach to process…” dialog box and that it is selected.
  • For example, attaching to iexplore.exe but selecting “T-SQL” as your attachment type will obviously not let you hit any Silverlight breakpoints.

One other solution: install Silverlight Tools for Visual Studio 2008 SP1 (again)

For me, the biggest single problem is flatly ensuring you have installed the Silverlight Tools installed after everything else.

With all the various installers, it’s actually possible to end-up with a fully functional Silverlight development environment, including all the GUI clues, but with debugging being an impossibility. How to diagnose it? Check if you have the Silverlight option in the debugging types:

Without this type listed in the “Attach to process…” dialog, Visual Studio will give you no other clue that it won’t be able to hit breakpoints – even if you have “Silverlight” listed as a debugging option in the website properties.

Installing the Silverlight Tools for Visual Studio SP1 ensures this is in place. Then you can go back through the other items in the checklist

Some useful related links:

• Silverlight Tools for Visual Studio SP1 mentioned above (http://go.microsoft.com/fwlink/?LinkID=143571)
  • Alternatively you can download the standalone Silverlight SDK from http://go.microsoft.com/fwlink/?LinkID=157102.
• Silverlight Toolkit (http://silverlight.codeplex.com)
  • Not to be confused with the required developer tools, this is an awesome open-source set of examples to get your started with Silverlight.
• Silverlight + WCF RIA Services (http://www.silverlight.net/riaservices/)
• The “Getting Started” page from Microsoft (http://www.silverlight.net/getstarted/)

Thanks to Microsoft and Xeed, I’m heading to the MIX10 conference which is running March 15th-17th.

I’m a little concerned by something though: everyone who I’ve told I’m in Las Vegas for a whole week has reacted with horror, sympathy and commiserations. A few days is fine, apparently – because you come home to recover. A whole week? Not so much apparently. Great! I’m there from the 12th till the 20th because actually, thanks to lastminute.com, I got a direct flight and a cheaper package than if I was only going for a few days. Now I’m starting to wonder if it would have been better to go for less time and pay to come home and sleep…

Anyways, on to MIX. One reason I’m looking forward to it (and maybe one reason why Microsoft have invited me) is I’ve already been working on a Silverlight UI for media services as part of the MimeCloud project Pete and I have set up. I’m looking at some exciting ways of getting that into a handy package on a mobile device for reporters and journalists in the field, having made a bet that Silverlight would be on WPS7. I’ll also be playing a role in giving people an introduction to Umbraco CMS, at both the free pre-party dinner and at almost every other thinkable opportunity.

Here’s some of the sessions amongst many that grab my attention, give me a shout if you’re going to be there!

SILVERLIGHT PERFORMANCE ON WINDOWS PHONE

BUILDING LARGE-SCALE, DATA-CENTRIC APPLICATIONS WITH SILVERLIGHT

CONNECTING YOUR APPLICATIONS IN THE CLOUD WITH WINDOWS AZURE APPFABRIC

THE HAAHA SHOW: MICROSOFT ASP.NET MVC SECURITY WITH HAACK AND HANSELMAN

WHAT’S NEW IN MICROSOFT ASP.NET MVC 2

BUILDING AND DEPLOYING WINDOWS AZURE-BASED APPLICATIONS WITH MICROSOFT VISUAL STUDIO 2010

p.s. I forgot to say – my good friend, ex colleague and MimeCloud partner Pete Miller is going too so come find us if you’re interested in our open-source digital asset manager

Doesn’t sound like the normal kind of Microsoft connection, does it? “Free and open-source”. But that’s exactly what I’ve been working on recently with them on a project to which I alluded already.

It’s about time I blogged about it.

I’m helping out in my spare time (which is thinning out amongst moving house and offices) on a collaboration between Microsoft and some great guys who are big in the Umbraco UK community. The project has come about quite by chance, for me anyway. I simply got a Skype IM from Warren Buckley asking if I could make a meeting at Microsoft about Umbraco ideas, and that was about it.

So, invite accepted and time having passed, I turned up at the MS offices in London with some concepts I’d thought up on the tube. Warren explains in his blogpost how the meeting came about, through various fortuitous connections, but I had come late to the party and didn’t really know much – other than Microsoft wanted to fund some new Umbraco packages.

I had been expecting this to be a secretive affair, keeping my ideas amongst myself and the Umbraco guys before sharing properly with the Big Corporate Giant – I mean, this was Microsoft, right? The Redmond behemoth that’ll sell your gran without asking and then charge you a fee?

Well, as it turned out I was a bit presumptive. Myself, Warren, Darren Ferguson, Tim Saunders, and Adam Shallcross all piled into a very funky meeting room and sat wondering where this would go. Will Coleman, Platform Strategy Advisor, and Mark Quirk (UK Head of Technology, Developer & Platform Group), proceeded to win us over in about 5 minutes flat.

As Will described, Microsoft wants to shake off some of its shackles and get lightfooted, and make genuinely friendly connections with the open-source communities. Starting with Wordpress, Drupal and most importantly Umbraco, they want to help by doing some seed funding of plugins and packages. The results and code can be released back into the community and, in a somewhat fair exchange, more people get a chance to see a new side to what the MS toolset can do.

So what has happened since?

Well, at that and subsequent meetings we agreed on a core set of ideas and packages on which to focus. Darren has done some great work using Silverlight to make Umbraco’s media library more accessible, and I’ve worked on its architecture. I also invited Pete Miller into the fray too so we can both help bring some of the experience from Condé Nast and our MimeCloud project where media is concerned. Tim and Ismail are working on an interesting combination of a key-value table datatype in Umbraco, together with a Silverlight UI for data manipulation. And there are a few other ideas up our sleeves to finish this month.

One aspect of building up awareness around this project is that we’ll all be blogging regularly and Microsoft will be spreading some link love far and wide. We’ll also be doing some screencast / video posts in the coming weeks around how we’ve used the tools.

I think it’s a nice project for Umbraco. A few new packages will appear on the Our repository, and Umbraco gets another pat on the back from a huge brand.

One other thing: Will is also likely to come along to CodeGarden10, as part of a Microsoft attendance there, which will be cool. He’s a nice bloke and I’m sure will really look forward to everyone at the conference doing what we did incessantly: jibe him about Google Docs and ask him for free copies of Windows…

The other week I had an idea on the Underground whilst crammed in during rush hour. I was face-to-face with a stinker of some fella’s armpit and my brain decided to change the subject.

I started thinking of Umbraco package ideas.

Just to be clear, my inner monologue doesn’t always go “Visual Studio! To the bat cave!” when faced with a mass of people – I have an excuse: I’m working on a Microsoft project at the moment, and the goal is to help the Umbraco community by working on some specific free packages. Every chance I’ve had, I’ve been thinking of exactly what I’d like to do to extend Umbraco with Microsoft products, especially with the focus of giving some useful tools to developers and end-users.

Not all of these ideas were any good, of course. And one was for Wordpress, so that was out. Awkward. Some of them were also a little out of scope too, focussing on Microsoft products that are already selling just fine for example. But it has definitely been refreshing to take a step back from 4.1 and start thinking around package development and “products”.

The thing is, some of those ideas that were out of scope still have legs even if there isn’t yet a budget. So in my spare time I thought I’d get stuck into my favourite.

I’ve started work on a Visual Studio 2008/2010 add-in which allows developers to configure Umbraco from within the IDE.

In particular, it will allow for the creation and editing of the document type structure, without switching into the browser. It’s coming on quicker than I thought and I should have something in March to release as a first beta.

With a bit of luck I’ll be able to get some MS help one way or the other, and I know Benjamin is keen on collaborating too as he’s been thinking along similar lines, so watch this space – Umbraco could be about to get even quicker from 0 to website.

I still reckon that Wordpress idea was cool though: “a plugin to replace all of Wordpress with basically something else”. Cracker.

Expected Beta 1: End March 2010

Today Pete and I reached a milestone with a project we’ve been working on in our spare time for a while. Within the last week we’ve decided to go public and will be spreading the word far and wide over the next month.

We got the CodePlex project sorted, and first checkin batch done for our open-source digital asset management system called MimeCloud.

The link to it on CodePlex is at mimecloud.codeplex.com, the mimecloud.com domains redirect there for now.

It’s not finishet yet; we’ve a fair few weeks to go until the first Beta, and we’re just starting to invite collaborators now that the core architecture is formed. But it’s going to be awesome, and I hope a lot of people will benefit from it. For a start, we’re going to a build a package so you can use it in Umbraco CMS

Here are a few snippets from the project homepage:

What is MimeCloud?

• MimeCloud is an open-source media asset library built for lightning speed and scalability
• You can use MimeCloud to:
  • Manage the images and video on your blog or website (Wordpress or Umbraco? No problem!)
  • Replace the image management module of your website’s CMS with something more powerful, extensible, and easier to use
  • Manage usage rights information on your digital media archive (publishers? Hello!)
  • Attach metadata in multiple languages to the same image, video or asset
  • Quickly upload 1 or 1000 assets in a batch, with resizing and thumbnail generation on the fly
• The project was founded by Alex Norcliffe and Peter Miller

What platform does MimeCloud use?

• MimeCloud builds against Microsoft .NET 3.5SP1, and consists of the following pillars:
  • Robust datalayer built on Microsoft SQL Server and Lucene.net
  • Extensible API in both WCF and RESTful forms exposed using IIS
  • Several clients including a full-featured Silverlight image manipulation and batch upload tool, and an ASP.NET MVC administration tool for basic upload and library searching
  • Storage of assets can be via FTP, HTTP upload, Amazon S3, Microsoft Azure – whatever you prefer

What’s a Media Library? Is this Digital Asset Management? I just want to upload images to my site

• No worries – just because MimeCloud can cope with millions of images and videos doesn’t mean it’s difficult to use, or resource intensive.
• You can have MimeCloud installed on a simple blog in minutes, and use it every now and then, and it will still have cost you nothing (unless you want to donate)

Does it only handle images?

• Not at all! We called it MimeCloud for a reason – the entire system is pluggable and supports anything describable by a MIME type. Out of the box we have support for images, videos, PDFs, Microsoft Office, and if there is a filetype you need that isn’t yet supported you can write a provider to tell MimeCloud how to handle it.

I finally got my Descriptive CacheManager Framework up on CodePlex.

I devised this in 2007 as part of a personal project, it’s code that I’ve maintained since, and it’s been in use on very high traffic sites of over 40 million pageviews per month in the past three years – so it’s pretty battle-hardened.

It’s 2010 and the time has come to share the code with the community!

What’s it for?

The Descriptive CacheManager Framework allows you to take a simple, different, and more effective method of caching items in ASP.NET. The developer defines descriptive categories to decorate their objects, rather than arbitrary timeout values, which are then used by the framework to determine an overall policy based on its knowledge of all items under its supervision.

It can also:

• Replicate cache items to other web servers in your load balanced setup (even those outside of the same network)
• Allow cache items to survive application restarts

All without replacing the famously good underlying ASP.NET cache itself.

The CodePlex project for it is here. Check it out and let me know what you think!

I’ll be doing a series of posts here and updates to the CodePlex project with demonstration configuration files in the coming week. Do get in touch if you want to know more information!

Today was cracking for a number of reasons: I got a bunch of stuff checked-in to the Umbraco 4.1 branch, made it to Umbraco’s 5th birthday event on the Thames in London, met up with a tonne of interesting people all variously involved in implementing Umbraco solutions around the UK, and did a talk with Pete Miller on scalability and cloud hosting.

The event itself was, by all accounts, a great success. Adam Shalcross (Cogworks) and Warren Buckley (Xeed) did a great job of organising it all, so much so that the venue had to be changed to fit the amount of people who wanted to go.

Here’s the video of our talk. The first part of my intro I think I was still recovering from 4 days’ food poisoning on which I’ll blame my propensity for saying “erm”, but I like to think I settled down eventually…

Alex Norcliffe & Peter Miller from CondeNast – Cloud computing & scalability from Paul Marden on Vimeo.