Rupert Rawnsley's WebLog

The Annotated URL (aURL)

2020-03-19T12:15:00.000+00:00

Here is a modest proposal for a URL annotation scheme that is useful when downloading static resources via URLs and offers a robust alternative to HTTP Cache-Control.

We have used this system for many years to reliably transfer terrabytes of data to hundreds of thousands of devices around the world. It could also be integrated into web browsers to significantly reduce the number of HTTP requests.

Specification

Annotated URL (aURL) parameters let you check cached files without making a new web request:

<url>?date=<unixtime>&size=<bytes>&md5=<hash>

For example:

https://www.google.com/favicon.ico?date=1481158857&size=5430&md5=f3418a443e7d841097c714d69ec4bcb8

Where:

&date lets you check if the remote file is newer than the local cached file and needs to be downloaded again.
&size gives a trivial check of file integrity without needing to hash the file and tells you the size before the request begins.
&md5 is between 1 and 32 hex nybbles of the MD5 file hash where increasing character count gives increasing confidence of integrity.

All aURL parameters are optional, but all three are recommended because URL characters are cheap and web requests are expensive.

aURLs can also be used to verify the integrity of a newly downloaded file with no additional information.

aURLs are CDN friendly because they naturally force the edge nodes to fetch modified resources.

aURLs could be generated automatically by publishing platforms like WordPress.

Love and the Singularity

2013-03-29T21:40:00.001+00:00

One of the milestones of the singularity is a computer capable of passing the Turing test: can you tell the difference between a computer and a real human using only text messages? It manages to be both subjective and definitive in that you cannot continue to maintain you prefer humans to machines if you can no longer tell them apart, much like the Pepsi Challenge.

In the context of the singularity, the Turing test is a canary in the coalmine. It is not practical as such, instead it signifies that computers have stormed the last fortress of human uniqueness (interestingly a synonym of singularity), and that they are now likely to be capable of our most precious gift: original thought.

But where is love? That most powerful part of the human experience and the driver for so much of what we do (both good and bad). A computer may become powerful and creative enough to cure malaria, but what is its motivation to do so? A human scientist does so for a mixture of love for his fellow humans, a love of the terrible beauty of biology, and a love of doing something he or she is good at. I expect all of these things are an important part of the creative process, possibly even essential, but I am not suggesting the capacity for love is somehow uniquely human and that this will prevent the singularity - it's not that kind of essay - I'm simply pointing out that truly intelligent computers may require the capability for love. Personally I find it harder to imagine a computer writing an ode than one beating me at chess; a feat already achieved by my treacherous cell phone.

If computers could love humans, or even if they couldn't, could we love them? This has significant consequences and shares some of the traits of the Turing test: love is the ultimate subjective emotion, but loving a computer would be a definitive step and a harbinger of societal change. It is hard to feel that computers are soulless automatons bent on destroying humanity if you are in love with one.

There are many types of love of course. Some of us already love our computers as we might love a treasured slide rule or sports car. It is not hard to imagine loving a computer as we love a pet: a condescending love of unequals. It is conceivable that we could love one as we love a colleague possessed of great insight and ability. But could we love a computer as we love a friend, a child, or a partner? If a computer were both lovable and orders or magnitude superior to us, would we not love them as gods? Each of these represents a different level of acceptance of the singularity and they represent increasing levels of disruption.

It is argued that a great disparity between human and machine will not arise because humans will be augmenting their own capabilities to match those of the artificial intelligences, and for the individual this may be viable, but no man is an island. For instance, a good marriage depends on growing together, but if your wife suddenly became a million times smarter than you, it's a good bet you would struggle with pillow talk. So do you only "upgrade" together? What about your children and your friends? These are complicated questions that could cause real pain during the constant revolution the singularity promises, but without love at the heart of it, it would truly be post-human.

If I speak in the tongues of men or of angels, but do not have love, I am only a resounding gong or a clanging cymbal. If I have the gift of prophecy and can fathom all mysteries and all knowledge, and if I have a faith that can move mountains, but do not have love, I am nothing. If I give all I possess to the poor and give over my body to hardship that I may boast, but do not have love, I gain nothing.

Android Emulation on x86 Machines

2012-03-23T11:26:00.000+00:00

It was news to me that Android doesn't just work on ARM processors, but apparently it has been running on x86 machines for a while.

Now thanks to some wizardry from Intel, the performance has been significantly improved by taking advantage of virtualization. Although x86-based Android devices aren't going to take over the world anytime soon, this does mean that we now have a viable Android emulator to use on the desktop.

Installation

The instructions are here and I suggest you follow them all. The only sticking point is if your machine supports Virtualization Technology or not. You can find that out by checking Intel's list or running their processor information tool. Even my budget ACER i3 laptop has it:

Then you need to run the Android SDK Manager and download version 17 of Android SDK Tools and the x86 system image:

One limitation is that the only AVD image available at the moment is for Android 2.3.3, but (depressingly perhaps) that covers 95% the devices out there. There is also an Intel configuration tool (in the Extras section) that you must install and run (see the aforementioned instructions). Note that the virtualization manager is memory hungry (I recommend a 2GB allocation), but RAM is so cheap that maxing out my laptop to 8GB only cost about £30 (it's the best investment you can make).

When you next open Android AVD Manager, you should be able to create a new x86-based VM:

Configure to your hearts content and launch...

You should find it a much faster experience than the ARM emulator and identical for Java-based applications. I'm not sure if NDK applications would work the same - comments welcome!

The possibility of developing applications completely on one machine really lowers the barrier to entry.

Android Permissions - Protection Levels

2011-11-29T11:22:00.001+00:00

Android applications declare the permissions they are likely to require in their manifest (a short file that describes the contents of the 'package'). This allows the system to sandbox them from critical resources and gives the user some indication of what havoc they might reap. That's the theory at least, but the first time I installed an application and read the permissions page I had no idea what they were on about! Clearly this system needs to be changed, but that is not what I want to talk about today.

As an application writer I need to know the protection level of these permissions, i.e. which of these permissions are normal (can cause the user no real harm), dangerous (might require a greater level of trust, such as the ability to read SMS messages), signature (only granted to applications that are signed by the people who built the OS), signatureOrSystem (like signature, but also allowed if they have been pre-installed in a system folder). I was surprised to find no easy reference for this in the documentation, but I did find the relevant information in the source.

You can of course probe the android package itself for this information, which is useful if you don't have access to the particular version of Android you are running. Here is some code that does just that:


// Get the permissions for the core android package

PackageInfo packageInfo = getPackageManager().getPackageInfo("android", PackageManager.GET_PERMISSIONS);

if (packageInfo.permissions != null) {

  // For each defined permission

  for (PermissionInfo permission : packageInfo.permissions) {

    // Dump permission info

    String protectionLevel;

    switch(permission.protectionLevel) {

    case PermissionInfo.PROTECTION_NORMAL : protectionLevel = "normal"; break;

    case PermissionInfo.PROTECTION_DANGEROUS : protectionLevel = "dangerous"; break;

    case PermissionInfo.PROTECTION_SIGNATURE : protectionLevel = "signature"; break;

    case PermissionInfo.PROTECTION_SIGNATURE_OR_SYSTEM : protectionLevel = "signatureOrSystem"; break;

    default : protectionLevel = "<unknown>"; break;

    }

    Log.i("PermissionCheck", permission.name + " " + protectionLevel);

  }

}

...and here are the results in case you need to know them at a glance...

Permission	Protection Level
android.intent.category.MASTER_CLEAR.permission.C2D_MESSAGE	signature
android.permission.ACCESS_CACHE_FILESYSTEM	signatureOrSystem
android.permission.ACCESS_CHECKIN_PROPERTIES	signatureOrSystem
android.permission.ACCESS_COARSE_LOCATION	dangerous
android.permission.ACCESS_FINE_LOCATION	dangerous
android.permission.ACCESS_LOCATION_EXTRA_COMMANDS	normal
android.permission.ACCESS_MOCK_LOCATION	dangerous
android.permission.ACCESS_NETWORK_STATE	normal
android.permission.ACCESS_SURFACE_FLINGER	signature
android.permission.ACCESS_WIFI_STATE	normal
android.permission.ACCOUNT_MANAGER	signature
android.permission.ASEC_ACCESS	signature
android.permission.ASEC_CREATE	signature
android.permission.ASEC_DESTROY	signature
android.permission.ASEC_MOUNT_UNMOUNT	signature
android.permission.ASEC_RENAME	signature
android.permission.AUTHENTICATE_ACCOUNTS	dangerous
android.permission.BACKUP	signatureOrSystem
android.permission.BATTERY_STATS	normal
android.permission.BIND_APPWIDGET	signatureOrSystem
android.permission.BIND_DEVICE_ADMIN	signature
android.permission.BIND_INPUT_METHOD	signature
android.permission.BIND_WALLPAPER	signatureOrSystem
android.permission.BLUETOOTH	dangerous
android.permission.BLUETOOTH_ADMIN	dangerous
android.permission.BRICK	signature
android.permission.BROADCAST_PACKAGE_REMOVED	signature
android.permission.BROADCAST_SMS	signature
android.permission.BROADCAST_STICKY	normal
android.permission.BROADCAST_WAP_PUSH	signature
android.permission.CALL_PHONE	dangerous
android.permission.CALL_PRIVILEGED	signatureOrSystem
android.permission.CAMERA	dangerous
android.permission.CHANGE_BACKGROUND_DATA_SETTING	signature
android.permission.CHANGE_COMPONENT_ENABLED_STATE	signature
android.permission.CHANGE_CONFIGURATION	dangerous
android.permission.CHANGE_NETWORK_STATE	dangerous
android.permission.CHANGE_WIFI_MULTICAST_STATE	dangerous
android.permission.CHANGE_WIFI_STATE	dangerous
android.permission.CLEAR_APP_CACHE	dangerous
android.permission.CLEAR_APP_USER_DATA	signature
android.permission.CONTROL_LOCATION_UPDATES	signatureOrSystem
android.permission.COPY_PROTECTED_DATA	signature
android.permission.DELETE_CACHE_FILES	signatureOrSystem
android.permission.DELETE_PACKAGES	signatureOrSystem
android.permission.DEVICE_POWER	signature
android.permission.DIAGNOSTIC	signature
android.permission.DISABLE_KEYGUARD	normal
android.permission.DUMP	dangerous
android.permission.EXPAND_STATUS_BAR	normal
android.permission.FACTORY_TEST	signature
android.permission.FLASHLIGHT	normal
android.permission.FORCE_BACK	signature
android.permission.FORCE_STOP_PACKAGES	signature
android.permission.GET_ACCOUNTS	normal
android.permission.GET_PACKAGE_SIZE	normal
android.permission.GET_TASKS	dangerous
android.permission.GLOBAL_SEARCH	signatureOrSystem
android.permission.GLOBAL_SEARCH_CONTROL	signature
android.permission.HARDWARE_TEST	signature
android.permission.INJECT_EVENTS	signature
android.permission.INSTALL_LOCATION_PROVIDER	signatureOrSystem
android.permission.INSTALL_PACKAGES	signatureOrSystem
android.permission.INTERNAL_SYSTEM_WINDOW	signature
android.permission.INTERNET	dangerous
android.permission.KILL_BACKGROUND_PROCESSES	normal
android.permission.MANAGE_ACCOUNTS	dangerous
android.permission.MANAGE_APP_TOKENS	signature
android.permission.MASTER_CLEAR	signatureOrSystem
android.permission.MODIFY_AUDIO_SETTINGS	dangerous
android.permission.MODIFY_PHONE_STATE	dangerous
android.permission.MOUNT_FORMAT_FILESYSTEMS	dangerous
android.permission.MOUNT_UNMOUNT_FILESYSTEMS	dangerous
android.permission.MOVE_PACKAGE	signatureOrSystem
android.permission.PACKAGE_USAGE_STATS	signature
android.permission.PERFORM_CDMA_PROVISIONING	signatureOrSystem
android.permission.PERSISTENT_ACTIVITY	dangerous
android.permission.PROCESS_OUTGOING_CALLS	dangerous
android.permission.READ_CALENDAR	dangerous
android.permission.READ_CONTACTS	dangerous
android.permission.READ_FRAME_BUFFER	signature
android.permission.READ_INPUT_STATE	signature
android.permission.READ_LOGS	dangerous
android.permission.READ_OWNER_DATA	dangerous
android.permission.READ_PHONE_STATE	dangerous
android.permission.READ_SMS	dangerous
android.permission.READ_SYNC_SETTINGS	normal
android.permission.READ_SYNC_STATS	normal
android.permission.READ_USER_DICTIONARY	dangerous
android.permission.REBOOT	signatureOrSystem
android.permission.RECEIVE_BOOT_COMPLETED	normal
android.permission.RECEIVE_MMS	dangerous
android.permission.RECEIVE_SMS	dangerous
android.permission.RECEIVE_WAP_PUSH	dangerous
android.permission.RECORD_AUDIO	dangerous
android.permission.REORDER_TASKS	dangerous
android.permission.RESTART_PACKAGES	normal
android.permission.SEND_SMS	dangerous
android.permission.SET_ACTIVITY_WATCHER	signature
android.permission.SET_ALWAYS_FINISH	dangerous
android.permission.SET_ANIMATION_SCALE	dangerous
android.permission.SET_DEBUG_APP	dangerous
android.permission.SET_ORIENTATION	signature
android.permission.SET_PREFERRED_APPLICATIONS	signature
android.permission.SET_PROCESS_LIMIT	dangerous
android.permission.SET_TIME	signatureOrSystem
android.permission.SET_TIME_ZONE	dangerous
android.permission.SET_WALLPAPER	normal
android.permission.SET_WALLPAPER_COMPONENT	signatureOrSystem
android.permission.SET_WALLPAPER_HINTS	normal
android.permission.SHUTDOWN	signature
android.permission.SIGNAL_PERSISTENT_PROCESSES	dangerous
android.permission.STATUS_BAR	signatureOrSystem
android.permission.STOP_APP_SWITCHES	signature
android.permission.SUBSCRIBED_FEEDS_READ	normal
android.permission.SUBSCRIBED_FEEDS_WRITE	dangerous
android.permission.SYSTEM_ALERT_WINDOW	dangerous
android.permission.UPDATE_DEVICE_STATS	signature
android.permission.USE_CREDENTIALS	dangerous
android.permission.VIBRATE	normal
android.permission.WAKE_LOCK	dangerous
android.permission.WRITE_APN_SETTINGS	dangerous
android.permission.WRITE_CALENDAR	dangerous
android.permission.WRITE_CONTACTS	dangerous
android.permission.WRITE_EXTERNAL_STORAGE	dangerous
android.permission.WRITE_GSERVICES	signatureOrSystem
android.permission.WRITE_OWNER_DATA	dangerous
android.permission.WRITE_SECURE_SETTINGS	signatureOrSystem
android.permission.WRITE_SETTINGS	dangerous
android.permission.WRITE_SMS	dangerous
android.permission.WRITE_SYNC_SETTINGS	dangerous
android.permission.WRITE_USER_DICTIONARY	normal
com.android.browser.permission.READ_HISTORY_BOOKMARKS	dangerous
com.android.browser.permission.WRITE_HISTORY_BOOKMARKS	dangerous

Cloud Production

2011-06-30T13:24:00.005+01:00

I've always been fascinated by 3D printers and recently ordered one from the crowd-funded Huxley project.

This project has raised far more than the initial expectation, and has forced the company that runs it to increase it's production capabilities by an order of magnitude. Now because these printers can print parts for themselves, eMaker are reaching out to other 3D printer owners to help them cope with the demand.

Imagine a future where you order something online and rather than coming from a local warehouse it is manufactured in a local facility that can make anything. As long as the quality is acceptable you don't have to know or care where it came from. The benefits of this from a production and supply chain point of view are enormous and efficiencies in the supply chain would mean cheaper goods for consumers. Couple this with the environmental benefits (less transportation and waste) and you have a game-changing technology: Cloud Production.

Passpack - Online password management

2011-05-27T14:42:00.003+01:00

Did I bore you about Passpack yet? If not read on...

Passpack is a website that manages your passwords and other login details. It is simple to use and allows you to share passwords with colleagues and family members.

For instance, if you add a link to the login page for the site, it will auto-complete the login fields (using a bookmarklet).

I can also recommend the automatic password generation, which helps you avoid principal danger of password reuse.

It has a neat system whereby the passwords are decrypted locally in the browser using your security pass-phrase. This means that even the Passpack folks can't see your passwords. Of course this means you should keep you pass-phrase written down somewhere - I recommend keeping it with your will :-)

For extra simplicity, you can login with your ID from Google, Facebook, Twitter, or and OpenID provider. This doesn't help with the pass-phrase, but it stops you needing two passwords: one to login and one to decrypt your data.

Until true SSO is a reality, this makes identity management much simpler.

ShareSafe.TV

2011-05-05T11:48:00.004+01:00

I often want to share YouTube videos with my kids, but they are surrounded by links to other videos, which can often be unsuitable. ShareSafe.TV displays only the video you want to show and nothing else. Use their link generator or just add v/<video id> to the end of their URL.

Here is an example: http://www.safeshare.tv/v/YXM3wrIhcwY

MSTest and 64bit

2011-04-20T10:44:00.015+01:00

This post is about running MSTest for applications that target mixed platforms.

If you are lucky enough to be able to write your applications in pure .NET, then you may never encounter 32bit/64bit platform issues. However, if any dependent library or plug-in is compiled for a specific architecture, then your whole application must be run in that mode. This is why the default Window's Internet Explorer is still 32bit despite the 64bit version shipping since Vista: it has to be the same architecture as any legacy plug-ins. By contrast, Notepad doesn't have any plug-ins, so it can get away with being 64bit only.

My companies applications rely on many native libraries, which are obviously compiled for specific architectures (x86 and x64). Deploying an application for multiple target processors is a complex subject in itself that can be solved with a range of strategies from dynamic library linking to processor-specific installers, but however you deploy, your application will behave differently in these two different modes so they must both be tested.

For better or for worse, we use MSTest to control application quality. Since the release of Visual Studio 2010 this has been able to run in 64-bit mode as well as 32-bit mode, but there are certain subtleties that complicate the practical aspects of administering your tests.

To understand the problem, consider the way MSTest works: Testing is done using two programs MSTest.exe and QTAgent32.exe. MSTest is told what assemblies to load and it scans those assemblies (using reflection) to find any classes and methods annotated as tests using the various test attributes. To do this it must be able to load the assembly and all its dependent assemblies and because MSTest is a 32bit process, none of these assemblies can be exclusively 64bit. Once loaded, MSTest instructs QTAgent32 to run these tests, which means QTAgent32 must load the assemblies itself and execute the test methods, but because it is also a 32bit process it cannot load 64bit assemblies either.

In Visual Studio 2010 a new version of QTAgent32 was added called QTAgent.exe, which can run 64bit assemblies. This means that even though MSTest is still 32bit, QTAgent can execute in full 64bit mode so that pure .NET assemblies can now be tested in 32bit and 64bit mode. However, it still doesn't easily allow applications with mixed-mode assemblies to be tested in 64bit mode because they cannot be loaded by MSTest in the first place.

One interesting solution to this is to force MSTest.exe to be a 64bit application. This implies that MSTest is actually pure .NET code anyway, but has been forced to run in x86 mode. If you are going down this road, note that MSTest relies on various registry entries (HKLM\SOFTWARE\Microsoft\VisualStudio\10.0\EnterpriseTools\QualityTools\TestType and HKLM\SOFTWARE\Microsoft\VisualStudio\10.0\Licenses) to decide which extensions it can handle and what features are licensed for use, and that these are installed by default to the WOW6432Node registry "shadow" branch. To run in 64bit mode you must copy some of the registry entries over as well as editing the binaries themselves.

There is an alternative approach that doesn't involve editing executable files and local machine registry settings (which can be a pain across a large development team). Our application builds for two distinct platform targets x86 and x64 (note however that most assemblies are compiled as "Any CPU" except the ones that contain native code) and test projects are only built in the x86 solution configuration. This ensures that the code in their bin folder is 32bit compatible, and they can therefore be loaded into MSTest. Also, the tests are configured to run against the binaries in the actual application deployment folder rather than running in their binary folder using the new root folder feature of Visual Studio 2010:

In this case we have used an environment variable that signifies where to find the deployed binaries. Whether you do this or not, it seems to always want a full path for one reason or another, which can make supporting multiple development environments a challenge.

We make one of these test configurations for each target platform, remembering to change the Hosts section that controls 32bit and 64bit execution. Then we can run both configurations from the command line like this:

mstest /testsettings:WorkStation32.testrunconfig /testmetadata:SOLUTION.vsmdi

mstest /testsettings:WorkStation64.testrunconfig /testmetadata:SOLUTION.vsmdi

The trick here is that in both cases MSTest will load the 32bit binaries to decide what tests to run, but the different configuration files will control if QTAgent32 or QTAgent is used. Note that this cannot work with the /noisolation switch, because MSTest cannot host the 64bit binaries.

The disadvantage of running your unit tests against the deployment folder is that your tests are less "clean" and test failures could take longer to diagnose. The advantage is that the tests are being run on code as it will appear in the wild, which can include complex deployment features such as assembly obfuscation.

This system will work on development desktops and build servers. It may give the Visual Studio IDE pause for thought occasionally, but it is fundamentally compatible, which is one of the only real advantages of MSTest in the first place.

IBM Watson plays Jeopardy

2011-02-18T12:20:00.002+00:00

This really changes everything: http://www.viddler.com/explore/engadget/videos/2393/

The most amazing computer demonstration I have ever seen.

Microsoft's Click-to-Run and Office Automation

2010-05-12T14:47:00.003+01:00

Today's lesson: When Office is installed using Click-to-Run, it doesn't support automation.

We use Excel automation via C# in our application, and when testing against the new versions of Office we hit a bump in the road. Office Home and Business 2010 typically installs via Click-to-Run, which is designed to have a small footprint, and as such does not register itself for programmatic automation.

So when you hear this sound: "Retrieving the COM class factory for component with CLSID {00024500-0000-0000-C000-000000000046} failed due to the following error: 80040154" even though Excel is apparently installed, you probably have this issue.

More here about Excel automation and the expected registry keys.

It is suggested that there will be an alternative MSI-based installer that presumably will not have this problem.

Vertical Alignment in CSS

2010-02-24T09:58:00.002+00:00

I know I should use CSS, but sometimes I fall off the wagon and use tables instead. This usually happens when I want to vertically align content. Here are some very clear hints and tips about vertical alignment that may help me kick the habit: http://phrogz.net/CSS/vertical-align/index.html

Content Delivery Networks

2010-01-03T20:31:00.003+00:00

My company has a very specific requirement: we need to get our application to any desktop in the world in less than three minutes. There are business drivers for this that I shall not go into; basically it is so that potential customers don’t get bored waiting for our application to install and run. We are currently failing to do that for all users, and we suspect we are losing customers because we fall at the first fence.

The Problem is Discovered

Our installer is about 50MB, which is not huge, but we have been seeing an enormous variation in deployment times to various parts of the world. Currently we use a UK-based hosting service with high symmetric bandwidth, but routine log analysis revealed that the install times for some users exceeded 10 minutes, and many did not complete. A quick web search revealed that this is a well known problem, so well known in fact that there are many commercial solutions that come under the generic title of Content Delivery Networks (CDNs). The big players are companies like Akami and Limelight, but I am allergic to companies that won’t tell you the price, and I suspect our needs are too modest to be worth their while addressing. There is however a new class of companies like GoGrid emerging and there are established hosting players like Amazon (with CloudFront) and Rackspace (using Limelights CDN network) who are offering CDNs. The new-kid-on-the-block is Microsoft, who beta-launched the Azure CDN solution just as my investigations began.

CDN, like all hosting, is a highly commodified product. There are certainly modest differences in terms of things like upload flexibility (Azure stinks), clever torrent links (Amazon S3 rocks), and general UI friendliness, but there were no showstoppers. The only really important metrics are speed, reliability, and cost. Cost was easy, everyone who didn’t make it clear on their website in the first two minutes was discarded (are you starting to understand our business drivers now?), and the remaining companies were all so cheap that it wasn’t worth worrying about. This is because we are talking about a very small amount of data 50MB x 100 installs per month = 5GB and the pricing is never more than about 25 cents per GB. These businesses are built for large streaming media and Flash media files, not for tiny desktop installers like ours.

Reliability next: we are not particularly concerned about reliability given that we are statistically unlikely to lose enough business in the difference between four nines and five nines to make it worth basing a decision on. Everybody can do four nines.

So that left speed, which comes in two flavors: latency and bandwidth. Latency is critical for that snappy website that puts your shop window in front of the customer in less than a few seconds (which is sometimes all you have). Incidentally, I didn’t come across any CDN webhosts, particularly ones that support ASP.NET, but you have to imagine it is coming from Azure. In our case, bandwidth was going to dominate so that is what we needed to know about.

During my research, I came across Ryan Kearney’s comparison of CDN providers. He gives a great round-up of the price and features of many of the providers, as well as some latency statistics for a handful of international locations. He was kind enough to host a file for my test rig on his Rackspace account, which was much appreciated.

So there are plenty of CDN providers, but very little information available to allow you to compare them. For instance, India and China are two very important markets for us, but what is the bandwidth to them from each of the providers? Clearly we needed to do some measurements.

The Game is Afoot

How do you measure the bandwidth of a host to every country in the world? Well, there are many companies that offer website monitoring and will alert you if your website goes down, some of these have international monitoring capabilities, and some of them have page download time statistics. However, to get an accurate picture of download speeds you need a fairly sizable file so that the bandwidth lag dominates other factors such as DNS resolution or server latency. Only one web monitoring service actually downloaded the whole file, allowing us to make an accurate estimate of bandwidth. They are WebSitePulse, and I could not have done this analysis without them. They have the most monitoring stations in the world, the most detailed statistics, and a 30 day free trial, which I used for this investigation. I highly recommend them to anyone looking for sophisticated, international, web site monitoring.

We created a test file called Test1MB.zip, which was a zip file that was truncated to exactly 1MB. A zip file is largely incompressible and the extension stops most servers from trying (actually few offer HTTP compression, which is a serious omission but beyond the scope of this post). This was mounted on multiple hosts and WebSitePulse was configured to download the files periodically. The WebSitePulse trial limits you to 20 monitor stations at a time (and excludes Auckland and Melbourne), and I didn’t have access to all of the hosts from the beginning, so the statistics are not done to laboratory standards. However, the statistical picture that emerges is reliable enough to allow business decisions to be made.

The Runners and Riders

Host	CDN Capable	Notes
RapidSwitch	No	Our current host and representative of good quality hosting in the UK.
Azure CDN	Yes	Still in beta, and we literally started using it the day it opened, so there were teething problems.
Rackspace	Yes	Huge player in hosting and cloud computing.
Amazon CloudFront	Yes	CDN at the front, Amazon’s S3 at the back. Nominally still in beta, but frankly charging for something means you must be judged as a commercial product.
Amazon S3	No	Our S3 hosting is in the US, so this is the standard candle for US-based cloud hosting.
GoGrid CDN	Yes	A high number of international points-of-presence, and more on the way.

Very few of the CDN companies offer free trials for some reason, but I think all are pay-as-you-go, which costs pennies for what we want. It took a bit of back-and-forth to get my GoGrid account set up, but their Twitter guy was great at fixing the problem once I made him aware of it. This meant that there are slightly less results for GoGrid. The whole trial ran for the best part of a month with roughly 15 minute poll times for every host. I had to change things around a bit as I went along to stay within the T&C’s of the WebSitePulse trial – you get $1000 to play with in total.

The following locations were monitored: Amsterdam, Bangalore, Beijing (2 monitors), Boston, Brisbane, Buenos Aires, Chicago, Dusseldorf, Guangzhou, Hong Kong, Houston, London, Los Angeles, Miami, Montreal, Mumbai, Munich, New York, Paris, San Francisco, Sao Paulo, Seattle, Shanghai, Singapore, Stockholm, Sydney (2 monitors), Tokyo, Toronto, Trumbull, Vancouver, Washington

The Results

The summary of the results is shown below:

Host	Uptime	Average 1MB DL Time (s)
GoGrid	100.00%	2.03
Rackspace CDN	100.00%	2.70
Amazon CloudFront	100.00%	4.46
Azure CDN	99.52%	4.67
Amazon S3	100.00%	5.04
RapidSwitch	99.98%	7.43

Here are the detailed results for all of the monitoring stations and hosts sorted into average download time order:

	GoGrid	Rackspace	Amazon CloudFront	Azure CDN	Amazon S3	RapidSwitch	Average
New York	0.12	0.19	0.24	1.00	0.50	1.39	0.57
Boston	0.17	0.24	0.42	1.06	0.54	1.31	0.62
Trumbull	0.16	0.39	0.55	1.38	0.50	1.36	0.72
Washington	0.21	0.36	0.98	1.04	0.34	1.80	0.79
Houston	0.24	0.34	0.30	0.73	1.20	2.20	0.84
Paris	0.23	0.31	0.40	2.39	1.78	0.24	0.89
Dusseldorf	0.20	0.29	0.18	3.08	1.92	0.27	0.99
Amsterdam	0.16	0.15	0.47	2.43	2.70	0.24	1.03
Chicago	0.05	0.19	1.95	1.02	1.60	1.48	1.05
San Francisco	0.30	0.30	0.22	1.58	1.83	2.31	1.09
London	0.15	0.37	0.40	3.68	1.89	0.18	1.11
Vancouver	0.15	0.41	0.23	1.57	1.64	2.69	1.12
Toronto	0.40	0.91	0.48	2.58	1.99	1.67	1.34
Seattle	0.16	0.31	0.21	2.15	1.64	3.66	1.36
Munich	0.63	0.29	0.31	4.17	3.06	0.67	1.52
Miami	0.35	4.06	0.70	1.72	0.83	2.25	1.65
Stockholm	0.71	0.20	0.82	4.48	4.82	0.67	1.95
Los Angeles	0.29	0.39	0.34	2.99	3.81	6.47	2.38
Sao Paulo	2.53	2.77	2.70	2.79	3.49	3.50	2.96
Brisbane	0.45	0.42	3.40	4.26	5.96	6.00	3.42
Tokyo	2.00	1.01	1.40	3.04	4.35	8.84	3.44
Sydney	1.17	1.25	3.19	4.15	5.81	5.78	3.56
Bangalore	3.32	0.76	1.75	4.61	8.33	2.66	3.57
Sydney 2	0.18	4.63	3.74	5.07	7.41	5.29	4.39
Montreal	1.01	1.57	1.46	12.44	2.14	8.40	4.51
Mumbai	4.23	2.96	2.00	4.87	10.40	3.70	4.69
Buenos Aires	5.80	7.03	6.41	7.60	6.25	5.33	6.40
Singapore	3.52	1.27	3.24	9.66	8.67	13.62	6.66
Hong Kong	1.24	1.76	1.24	7.65	9.28	26.92	8.02
Beijing 2	5.72	8.23	11.62	7.95	11.20	17.53	10.38
Guangzhou	5.43	10.99	19.48	8.23	10.30	10.91	10.89
Beijing	8.50	14.33	23.92	13.37	16.29	71.55	24.66
Shanghai	17.32	20.56	52.27	19.37	23.83	24.16	26.25
Average	2.03	2.70	4.46	4.67	5.04	7.43	4.39

Here are the raw stats if you would like to do any further analysis of your own.

Conclusions

Clearly GoGrid and Rackspace are the best providers from the hosts tested. GoGrid has the best average performance and is unbeaten to almost all of the monitoring stations.

Asia is very badly served by all the hosts tested. Obviously there are dedicated hosting services for Asia, but the whole point of a CDN is that it is global. I expect partnerships are being drafted as I type.

Amazon S3 barely outperforms CloudFront on average, but peak download times per city are much better in some cases.

Montreal did much worse than I expected given that Canada is so well connected to the US.

Amazon and Azure CDN’s both perform equally well, although the uptime of Azure looks bad. Actually the Azure uptime was only really bad for the first few days, after that it was very good, so it is probably not a fair measure.

Did We Win?

Our original aim was to move 50MB in less than three minutes. Therefore our target time for 1MB is 180 / 50 = 3.6 seconds. Even with the fastest CDN host, we are still failing to meet this target for several cities. For Shanghai, we are a factor of five off. And of course this is before we get from where the monitoring stations are (which is probably a well connected hub) out to users at the network edge.

So big-iron can help us make significant improvements for very little effort and cost, but the war goes on. I might tell you how we finally win in a future post. Hint: we make the installer smaller.

Google AdSense Fraud

2010-01-03T15:30:00.001+00:00

As I mentioned in my last post, we stopped advertising through Google’s Content Network very soon after we started experimenting with AdSense when we thought we had detected significant fraud. Having not spent much money by this time, the amounts involved were relatively small (less than £100), but the fraction was high. We considered at least 1/3 of our clicks to be fraudulent: deliberately, criminally fraudulent.

Proud of our forensic IT skills, we rushed to Do-No-Evil Google to report our discovery. Our report contained the bare minimum of facts; we were so convinced it screamed fraud we did not bother to go into much more detail than the websites and the Click Through Rates (CTRs). After a week we had a polite and considered response that attempted to persuade us that this was not fraud. Clearly more evidence was required, so we put together a verbose description of the main points that alerted us, and sent it off in expectation of an apology if not a seat on the board for our fraud-busting smarts.

Sadly they refused / failed to be convinced and eventually we had to agree to disagree. Obviously we will continue to use Google for advertising – what choice is there for an online business? But we firmly believe they need to put their house in order. If 1/3 of their revenue is fraudulent they will lose consumer confidence and possibly face sanctions for complicity.

Below I have included the email chain. The name of the operative has been removed as the problem is systemic not personal. The key point, and one I should have put front-and-center (in CAPS perhaps?), is under Red Flag 2 – Set 2. Namely that a badly named, unlinked, and parked-domain website achieved ten times the CTR of Google’s own homepage.

I would very much like to hear from anyone with similar experiences, or from anyone (at Google or otherwise) who disagrees with any of the points we made. Our transaction volume is small and our market is niche, so I am aware we need more data to make a statistically significant conclusion.

Initial Inquiry

[From Rupert to Google on 5/11/09]

Nearly *one third* of all content based CPC placements were obvious click fraud.

Many of the sites have exactly the same content and have a CTR of 100% (or greater in one case!).

I find it incredible that the market leader cannot spot something like this automatically. The fraudsters are not even trying to disguise it.

[Information about the site URLs and the fraud period]

Google’s Response

Hello Rupert,

We have received your request for an invalid clicks investigation. Thank you for your patience while we reviewed your account. I apologize for our delayed response. I understand you are concerned about the quality of clicks you have accrued from certain sites in our content network.

We reviewed your account and can confirm clicks from these sites. However, we found that these clicks are valid, and there is no activity that suggests you have been charged for invalid clicks. The clicks charged fit a pattern of normal user behaviour. As part of our review, the team looked through dozens of data points--including IP addresses, IP blocks, geographic concentrations, network activity, browser patterns, click timings, and any proprietary signals. However, none of those suggest an automated attack, nor collusion from unethical users. The clicks accrued reflect normal user traffic.

Many of the sites that you listed are parked domain sites. A parked domain site is an undeveloped web page belonging to a domain name registrar or domain name holder. Our AdSense for domains programme places targeted AdWords ads on parked domain sites that are part of the Google Network.

Users are brought to parked domain sites when they enter the URL of an undeveloped web page in a browser's address bar.

We've found that AdWords ads displayed on parked domain sites receive clicks from well-qualified leads within the advertisers' markets. In general, we've noticed that the return on investment gained on these pages is equal to or better than that gained on other pages in the search and content networks. However, if you aren't satisfied with the value of the traffic, you can prevent your ads from showing on parked domain sites by using the Site and Category Exclusion tool. Learn how at https://adwords.google.com/support/bin/answer.py?answer=86695&hl=en_GB.

I hope that this information helps address your concern. Please let me assure you that your security is a top priority for Google, and we will continue to monitor all clicks on your ads to prevent abuse. Let us know if you have further questions or if we can be of any more assistance. For more information about steps we take to combat invalid click activity, please visit https://adwords.google.com/support/bin/answer.py?answer=6114&hl=en_US.

Sincerely,

<Google Employee>

The Ad Traffic Quality Team

Rupert’s More Detailed Description

Hi <Google Employee>,

Thank you for getting back to me. I'm afraid I am still doubtful of the validity of these sites. Allow me to illustrate my concerns with some examples:

Red Flag 1

There are 10 sites in the list with a 100% CTR (radiolluvia.com even has 200%):

Domain

Clicks

Impressions

CTR

radiolluvia.com

2

1

200.00%

net-ebooks.com

1

1

100.00%

umtsfree.net

1

1

100.00%

littleabout.com

1

1

100.00%

mtncareer.com

1

1

100.00%

jonefm.com

1

1

100.00%

radiobendele.com

1

1

100.00%

iphalloween.info

1

1

100.00%

pdfee.com

1

1

100.00%

rf-online.com

1

1

100.00%

None of them contain any relevant content (which would be fine - I understand AdSense can never be a science), but most of them consist only of AdSense links. Are you really suggesting that users happen upon a parked domain by typing the URL above into the browser and happen to be in the market for radio planning software (which is what we make)?

We don’t make a mass-market product so I would never expect a high CTR on content networks even when targeting radio engineers – only a minority of them are even actively seeking our type of tool.

Red Flag 2

There are at least three sets of pages that contain practically the same content and come from the same IP subrange.

Set 1: gsmsandwich.com, jonefm.com, keonong.com, mtncareer.com, rf-indo.com, smsgupsup.com. They all look like this:

Set 2: umtsfree.net, xlgprs.net, xlgprs.com, ir-hot.com. They all look like this:

I find it unlikely that in the space of one week, four people were using these sites as some sort of search or index portal (have they not heard of Google? J) and either searched to or browsed to our advert and found it relevant enough to click on. This was after only 254 impressions. Contrast this with Google’s own sponsored search results, which yielded only three hits for 2,051 impressions during the same period. If you really believe these statistics are true, surely you should buy this company immediately because they are 10 times better at advert placement than you are. Perhaps you should consider a smiling coed on the homepage?

Set 3: gamezerm.com, and radiobendele.com both have the same IP and look the same:

Can it be a coincidence that they have high CTR (50% and 100% respectively)?

In general, all of the sites in these sets have dubious registration details, often using the same registration anonymity service.

Red Flag 3

The click-through path is curious for the links from these sites. For instance, the umtsfree.net site links go through five 302 redirects before landing at the intended target. Here is the chain of URLs:

http://umtsfree.net/forward302.aspx?epc=eWpDPeDkCn7%2fPAWjJDHHizukuSQO4Z0sDU8KfdC9FQ%2b5yWjWwcxv5hXcA5nQpS0OqiEn07sYHTHFe%2fX9vFjDwUcSW01%2bS4WPEL0m7%2fX3z100tRxVe1Mg2zzaXK862vPp7hIJBvAoVV9DPRmnuG%2fkV0w5tbowrxB4AbcTtxa0Bsr%2fCztN7vTUOE0hGYneCC9V5jEY3PRhY5SAeWBCuCp7NzUBODKuSrYrmWbY4g3PHs9mBH08pqUSaY75VuOBggtVC6D5WjIQEZuFNJS10GQ6Bu%2f0JpRTb3xpAWZf4bPOguFyT3zwx6udcQe031GVCTob%2bAk5n3HzuAg2AOTMKncWxG%2bPl6vLUW4DWYQil2ZmY2ILRGYWgOHHAfIlNM1AHowYkUvb%2bBrYHbEQgD6PTID5%2fuaj3OsxHAwLVlhrL3uuu1S3zI4g9mOUab2fnM8yr%2brQmzu2a6UmflkA8s6PaAElxBNg9kZnshsusDIleugD02G6c%2bCTybRaQ0D1IYTOcfyeJLFDejgK2GqObGWs9Nm6J32886U0STHIAz75%2f%2b2snbAtAJQT48cwhAH%2fNQ%2fpaJiHkSON1cIxd7oFroekPJ8iyDhbYZ3VP1TJ0Z7HoHj6HKjeDaemj6LFb1Le0uwGKeLe%2bKc6LxdhYBtjD%2bXnGi0LkIajkmbqWe73rvyLNoRhtd1SsqgDT7wxMUBmPIaisppXnfp%2b8%2ftjiL8R9LU6QFh%2f8%2b4aTQGsrMOTyj55aZ4hx8l3UJck6utVoeVax7%2bOQACwyLYoyyvI4ml9DVsz%2f9Mh4WnmfFdVgPEqIxmJ%2bwnNIlKzX31GePRuLmgLHdeItkfnMPnUWIA8FB485RmGfrEwbF07d7v5JLcYuL4V62CKyW8dl3m89EvsbxcbOTOQN85CQsa5fKdgUZaZ2j1kFgi7Oj9J3MJ10oCQ5OjkZnoBWXZPFsfGxQTxg%2fGxh9k18jep%2b5sesYVB1jRYuej3ptGZBGivoDvEkFR%2fpxCbqB494irMYSWLmmx8c%2frOVZYeIe3XV9P7cBJ4da%2bcrLgJneN2nhKCOX0BDZsw%2bR1L93vZ6LgjgvFgolOFTVpkeF12ecpQWJg5jzm0AnoUhGdj%2fXzFJoJbgaxLnvBsFGql9%2f%2bYyJqMr8URZuovttYKmDemHYS0

http://rc12.overture.com/d/sr/?xargs=15KPjg141SnJamwr%2DocLXBROWAylwaxca58cluD5l4GtZf5iMxXOV4aaTCm8dxTOVxv1PdzPSW%5FqYSL%5FT5kPOJGweKQVWJGuXpjdLJxYw6Nq2jUNEbsYRzy%2DLvmIZGOX0E2laEOd%2D5mO7acZdRD05mjddAwByR%2D%5Flqw8yzxu4IQevVig0sskqFc5Z17tQp9bnAXOx7TLome97vhXfFfZwQ%2D%2DxDke%2DgSygTLyyj4WYa9VeHJi58obDIYo0L3ZbKzoLLOKeswIYJfRXG%2DYe62VuOrU6t8txuN2zT3r4MzgFZJP%5F%2DIlWJ3Ulvvv%2DbgfDfP4074wP1CfzqVHz3dxM5PXU3E5OufGXnbWw99E%5FOfpRQIMSv2xOO

http://clickserve.uk.dartsearch.net/link/click?lid=43000000042332928&ds_s_kwgid=58000000000470369&ds_e_adid=8770229031&ds_e_matchtype=standard&ds_e_kwdid=86608522531&ds_e_kwgid=5777302919&ds_url_v=2

http://ad-emea.doubleclick.net/clk;160208746;22377034;b;u=ds&sv1=42332928&sv2=2009111264&sv3=84754;%3fhttp://www.marshallward.co.uk/?aff=yahoo?&affsrc=acquisition&cm_mmc=yahoo-_-Generic-_-Generic-Catalogue-Keywords---Broad-_-catalogs

http://fls.doubleclick.net/act;sit=530730;spot=1529997;~dc_rdr=?http%3A//www.marshallward.co.uk/%3Faff%3Dyahoo%3F%26affsrc%3Dacquisition%26cm_mmc%3Dyahoo-_-Generic-_-Generic-Catalogue-Keywords---Broad-_-catalogs

http://www.marshallward.co.uk/?aff=yahoo?&affsrc=acquisition&cm_mmc=yahoo-_-Generic-_-Generic-Catalogue-Keywords---Broad-_-catalogs

This would allow the site operator to monitor clicks, which is a legitimate thing to do. It is also something you would need if you wanted to monitor and reward agents clicking the links for you. I appreciate that this in itself is not a “smoking gun”.

Conclusion

I suggest that at least the sites in these sets are fraudulent. The fact that they evaded the data mining checks you mention suggests to me that they were generated by a network of geographically distributed agents. These might be humans, encouraged by a share of the AdSense revenue or they might be an autonomous bot-net with a smart pattern of click behaviour (chaotic perhaps). Frankly the click-pattern doesn’t seem that smart to me, so I think there is another type of fraudster out there that is much smarter and rarely hits the same advertiser twice, this would prevent detection by the advertiser, and would only be detectable by Google themselves.

Let me be clear that I have no reason to doubt the validity of sponsored search or Gmail content adverts and I continue to use these networks, but I have serious doubts about the public AdSense network. I also do not argue that it is still good value for money (even if 1/3 of it is click-fraud), but nobody likes to be ripped off and it is in both our interests to fix it.

What do we expect? We are not looking for refund on these clicks (frankly I have wasted more money typing this email), instead we would like Google to consider my arguments, and if they agree, to improve their detection process. Excluding these sites is a poor option as I will have to spend time every day weeding out spammers from our content network placements. I appreciate that there will always be a number of frauds that are impossible to detect algorithmically, and that you are locked into an arms race with the fraudsters, but there seems to be more you could be doing to improve automatic detection.

Best regards,

Rupert Rawnsley.

Google’s Response

Hello Rupert,

Thank you for your reply and for providing us with the additional information. We appreciate your patience as we work to resolve this issue.

I can confirm the information you have mentioned. Many of the sites in question seem to have the same templates and show only AdSense ads.

However, the clicks that you accrued from these sites are valid. As mentioned in our previous email, these sites are a part of the domain park network. The sites in question do not have any specific content, but are simply "parked" for interested users to purchase the site from the domain hosting company. Also, domain parked sites can be former functioning websites whose domain name contracts have expired. Since these sites are largely created for temporary purposes, the template used may be the same across several websites. This is the reason you may find the same images or the same layout across several of these sites.

Once an interested user purchases the site or renews the domain name registration, the site is automatically removed from the domain parked network by the hosting company. Parked domain sites offer users ads that are relevant to the text they entered. In addition, some parked domain sites include a search box, which allows users to further refine their search. We've found that AdWords ads displayed on parked domain sites receive clicks from well-qualified leads within the advertisers' markets.

In general, we've noticed that the return on investment gained on these pages is equal to or better than that gained on other pages in the search and content networks.

We are sorry to learn that you are disappointed with the quality of clicks you accrued from these sites. Please be assured that the clicks are valid.

We strongly suggest you consider using the site-and-category exclusion tool to prevent your ads from showing on the domain park network.

Thank you for your patience and understanding.

Sincerely,

<Google Employee>

The Ad Traffic Quality Team

Three Months With Google AdWords

2009-12-16T19:40:00.008+00:00

We recently changed our business plan and embraced online selling. We were obviously aware it existed before, we just didn't have anything that you could effectively sell in that way. Online selling means online advertising, and we all know that means Google. This gave us our first look at AdWords - Google's advertising OS - and we were impressed, but we soon discovered some unexpected things that you might find interesting.

The Anatomy of Google AdWords

AdWords has two distinct channels: search and content network. Search is the "Sponsored Links" section that you see at the side and top of search results. Content network is everywhere else that Google puts adverts, which is principally the AdSense network and GMail. By default, adverts are delivered across all channels and there are numerous analyses for seeing how they perform. We have been advertising with a very modest budget to a niche market for three months, but in that time we have had some eye-opening experiences.

AdWords is Expensive

You don't pay when Google shows an advert, only when someone clicks on it (a click-through). However, a click-through from the search page costs us £2.50 and from the content network it is about 80p. I assumed it would be much less than this, but then I have led a sheltered life. I am not saying it is not worth the money; it just costs more than I expected.

You Are Google's Bitch

As an online business, we rely on people being able to find us online (duh!). Either they are looking for us explicitly or are looking for the type of thing we sell. 70% of people will do this using Google's organic search, so if Google decide your site is suspect or undesirable in some way and decide to de-list you, your market just got a whole lot smaller. There is no discussion process, no independent authority to whom you can appeal, no cost to Google, and no way back. Sites can be demoted or completely blocked erroneously and without warning or explanation just for something as simple as putting light text on a light background (because it looks like you are trying to fool the GoogleBot into promoting your search rank). This is a dangerous monopoly and it needs to be addressed, and I had never really thought about it before, but it is not my central thesis today.

AdSense is Suspect

In the first week, most of our spending was going into the AdSense network, but when we looked at the target sites were disappointed by the placement choices. More seriously than that, we found strong evidence of click fraud. I don't want to divert from my central thesis at this point (it's coming I promise), so this will be the subject of my next posting. I only raise the point now to say that we suspended AdSense adverts, so all "content network" placements were on GMail only.

Search is Irrelevant

Here is the really interesting thing: In the three months we have been advertising we have had 20,000 impressions on the search pages but 1,000,000 impressions on GMail. Just 2% of the eye-time that we bought from Google came from search.

Assuming our figures are typical (and this is by no means certain given the niche nature of our business) it begs the question: why does Google stay in the search game? The most obvious answer is that people think, as we did, that search engines are the primary online advertising platform, so they take their marketing dollars to Google, only to have Google spend them elsewhere. Therefore, Google must maintain their dominance in the search game or companies will move their budgets elsewhere based on the incorrect assumption that less people would see their adverts.

Google is Vulnerable

It also implies Google is vulnerable. Ignoring AdSense for a moment (which is risky, but work with me here), GMail is not the biggest WebMail provider by a long chalk. Plucking the first numbers I could find off the web I see that GMail has about 90m users compared to Microsoft and Yahoo's 250m apiece (see why Yahoo is still a valuable asset now?).

It appears to me that in taking Google on at search, Microsoft and Yahoo are charging at the cape rather than the bull. Show people that you are the advertising channels of choice and hit Google where it hurts.

Bran Ferren - Leadership

2009-12-03T12:49:00.004+00:00

Bran Ferren is an imagineering alumnus and co-founder of the coolest company in the world (with the simplest website).

At two hours, this lecture is long, but it flies by (and I won't normally watch anything on the Internet longer than 4 minutes). Bran is so lucid and inspiring that his enthusiasm is infectious.

I was reminded of some old Feynman lectures I have seen, also to MIT students as I recall. Feynman's lecture was outside (it was the 60's) and the questions were about anti-gravity (it was the 60's), Ferren's lecture was inside and mentioned business and the military-industrial complex without being heckled (it wasn't the 60's). Whether this is progress or not is left as an exercise for the reader.

I can't remember who turned me on to this, so apologies for omitting credit. Probably BoingBoing.

WARNING: Not safe for people with pogonophobia!

ClickOnce Deployment Problems

2009-10-15T14:30:00.010+01:00

We have recently moved to ClickOnce for all our application deployment needs, and it is fantastic in so many ways, but there are still some scenarios where it fails. I will collect these together in one post and keep it up-to-date as new information becomes available. If anyone else has any ClickOnce war-stories, please add them to the comments.

Useful Links

MSDN Server and Client Configuration Issues in ClickOnce Deployments

ClickOnce and Setup & Deployment Projects MSDN forum.

RobinDotNet's Blog has some useful tips, including .NET 4.0 logging. Robin is an MVP and ClickOnce forum moderator.

Proxy Servers

UPDATE: Thanks to Robin for this link to a fix for the proxy issue.

If your customers are behind an authenticated proxy server, ClickOnce may fail to work. This is because the authentication settings are usually stored in Internet Explorer, but the ClickOnce deployment libraries (System.Deployment.Application) do not use them by default. The bug is documented here, but opinion varies about whether it should be fixed or not: Microsoft says it shouldn't, everybody else says it should.

When it goes wrong a typical error report looks like this:

PLATFORM VERSION INFO
Windows : 5.2.3790.131072 (Win32NT)
Common Language Runtime : 2.0.50727.3053
System.Deployment.dll : 2.0.50727.3053 (netfxsp.050727-3000)
mscorwks.dll : 2.0.50727.3053 (netfxsp.050727-3000)
dfdll.dll : 2.0.50727.3053 (netfxsp.050727-3000)
dfshim.dll : 2.0.50727.3053 (netfxsp.050727-3000)

SOURCES
Deployment url : http://overtureonline.com/Applications/Keima/Overture/Overture.application

ERROR SUMMARY
Below is a summary of the errors, details of these errors are listed later in the log.
* Activation of http://OvertureOnline.com/Applications/Keima/Overture/Overture.application resulted in exception. Following failure messages were detected:
+ Downloading http://overtureonline.com/Applications/Keima/Overture/Overture.application did not succeed.
+ The remote server returned an error: (407) Proxy Authentication Required.

COMPONENT STORE TRANSACTION FAILURE SUMMARY
No transaction error was detected.

WARNINGS
There were no warnings during this operation.

OPERATION PROGRESS STATUS
* [25/09/2009 15:59:15] : Activation of http://OvertureOnline.com/Applications/Keima/Overture/Overture.application has started.

ERROR DETAILS
Following errors were detected during this operation.
* [25/09/2009 15:59:16] System.Deployment.Application.DeploymentDownloadException (Unknown subtype)
- Downloading http://overtureonline.com/Applications/Keima/Overture/Overture.application did not succeed.
- Source: System.Deployment
- Stack trace:
at System.Deployment.Application.SystemNetDownloader.DownloadSingleFile(DownloadQueueItem next)
at System.Deployment.Application.SystemNetDownloader.DownloadAllFiles()
at System.Deployment.Application.FileDownloader.Download(SubscriptionState subState)
at System.Deployment.Application.DownloadManager.DownloadManifestAsRawFile(Uri& sourceUri, String targetPath, IDownloadNotification notification, DownloadOptions options, ServerInformation& serverInformation)
at System.Deployment.Application.DownloadManager.DownloadDeploymentManifestDirectBypass(SubscriptionStore subStore, Uri& sourceUri, TempFile& tempFile, SubscriptionState& subState, IDownloadNotification notification, DownloadOptions options, ServerInformation& serverInformation)
at System.Deployment.Application.DownloadManager.DownloadDeploymentManifestBypass(SubscriptionStore subStore, Uri& sourceUri, TempFile& tempFile, SubscriptionState& subState, IDownloadNotification notification, DownloadOptions options)
at System.Deployment.Application.ApplicationActivator.PerformDeploymentActivation(Uri activationUri, Boolean isShortcut, String textualSubId, String deploymentProviderUrlFromExtension, BrowserSettings browserSettings, String& errorPageUrl)
at System.Deployment.Application.ApplicationActivator.ActivateDeploymentWorker(Object state)
--- Inner Exception ---
System.Net.WebException
- The remote server returned an error: (407) Proxy Authentication Required.
- Source: System
- Stack trace:
at System.Net.HttpWebRequest.GetResponse()
at System.Deployment.Application.SystemNetDownloader.DownloadSingleFile(DownloadQueueItem next)

COMPONENT STORE TRANSACTION DETAILS
No transaction information is available.

A smart customer of ours actually found an answer: she ran an NTLM proxy, but we shouldn't expect all our customers to be computer geniuses.

The recommended solution is for the customer to edit their machine.config file - I can't see how that could possibly go badly ;-)

Internet Explorer Security

If the Internet Explorer security option "Run components not signed with Authenticode" option is set to "Disable", then your .application file will not work. The reason why would be clearer if the wording was "Run components not signed with Authenticode certificates that I trust", because even if your installer is actually signed, it is not necessarily a trusted publisher on the customers machine.

The error looks like this:

PLATFORM VERSION INFO
Windows : 6.0.6001.65536 (Win32NT)
Common Language Runtime : 4.0.20506.1
System.Deployment.dll : 4.0.20506.1 (Beta1.020506-0100)
clr.dll : 4.0.20506.1 (Beta1.020506-0100)
dfdll.dll : 4.0.20506.1 (Beta1.020506-0100)
dfshim.dll : 4.0.20428.1 (Beta1.020428-0100)

SOURCES
Deployment url : http://cadenza.keima.co.uk/Applications/Keima/Test/ConsoleApplicationClickOnce/ConsoleApplicationClickOnce.application
Server : Microsoft-IIS/7.0
X-Powered-By : ASP.NET

ERROR SUMMARY
Below is a summary of the errors, details of these errors are listed later in the log.
* Activation of http://cadenza.keima.co.uk/Applications/Keima/Test/ConsoleApplicationClickOnce/ConsoleApplicationClickOnce.application resulted in exception. Following failure messages were detected:
+ Your Web browser settings do not allow you to run unsigned applications.

COMPONENT STORE TRANSACTION FAILURE SUMMARY
No transaction error was detected.

WARNINGS
There were no warnings during this operation.

OPERATION PROGRESS STATUS
* [10/15/2009 3:21:06 PM] : Activation of http://cadenza.keima.co.uk/Applications/Keima/Test/ConsoleApplicationClickOnce/ConsoleApplicationClickOnce.application has started.

ERROR DETAILS
Following errors were detected during this operation.
* [10/15/2009 3:21:06 PM] System.Deployment.Application.InvalidDeploymentException (Manifest)
- Your Web browser settings do not allow you to run unsigned applications.
- Source: System.Deployment
- Stack trace:
at System.Deployment.Application.ApplicationActivator.BrowserSettings.Validate(String manifestPath)
at System.Deployment.Application.ApplicationActivator.PerformDeploymentActivation(Uri activationUri, Boolean isShortcut, String textualSubId, String deploymentProviderUrlFromExtension, BrowserSettings browserSettings, String& errorPageUrl)
at System.Deployment.Application.ApplicationActivator.ActivateDeploymentWorker(Object state)

COMPONENT STORE TRANSACTION DETAILS
No transaction information is available.

I don't know of any tidy workaround for this. If the client installs your certificate prior to product installation then it will work, but then it's no longer ClickOnce is it? One might argue that they asked for this by changing the setting, but it is more likely that their IT department did it and they know nothing about it. The irony is that if you run the bootstrap setup.exe, as you must from browsers like Chrome, then it works fine, but this means you are forcing clients to run a full-trust executable instead of a potentially low-trust .application file.

UPDATE: Robin points out in the comments below that this is an IE6 only issue.

Missing Local Files

A ClickOnce install puts files in a special folder called Apps (%localappdata%\Apps on Vista at least). I'm not sure the exact purpose of all of the folders and files, but I do know if you delete some of them it can cause web-launch failures. For instance, I installed a simple application, deleted the folder which held the application binary files and then tried to relaunch the application. This failed with the following message:

PLATFORM VERSION INFO
Windows : 6.0.6001.65536 (Win32NT)
Common Language Runtime : 4.0.20506.1
System.Deployment.dll : 4.0.20506.1 (Beta1.020506-0100)
clr.dll : 4.0.20506.1 (Beta1.020506-0100)
dfdll.dll : 4.0.20506.1 (Beta1.020506-0100)
dfshim.dll : 4.0.20428.1 (Beta1.020428-0100)

SOURCES
Deployment url : http://cadenza.keima.co.uk/Applications/Keima/Test/ConsoleApplicationClickOnce/ConsoleApplicationClickOnce.application
Server : Microsoft-IIS/7.0
X-Powered-By : ASP.NET

IDENTITIES
Deployment Identity : ConsoleApplicationClickOnce.application, Version=1.0.0.1, Culture=neutral, PublicKeyToken=39bbeb210852ebe6, processorArchitecture=msil

APPLICATION SUMMARY
* Online only application.

ERROR SUMMARY
Below is a summary of the errors, details of these errors are listed later in the log.
* Activation of http://cadenza.keima.co.uk/Applications/Keima/Test/ConsoleApplicationClickOnce/ConsoleApplicationClickOnce.application resulted in exception. Following failure messages were detected:
+ The directory name is invalid. (Exception from HRESULT: 0x8007010B)

COMPONENT STORE TRANSACTION FAILURE SUMMARY
No transaction error was detected.

WARNINGS
There were no warnings during this operation.

OPERATION PROGRESS STATUS
* [10/15/2009 3:32:28 PM] : Activation of http://cadenza.keima.co.uk/Applications/Keima/Test/ConsoleApplicationClickOnce/ConsoleApplicationClickOnce.application has started.
* [10/15/2009 3:32:28 PM] : Processing of deployment manifest has successfully completed.

ERROR DETAILS
Following errors were detected during this operation.
* [10/15/2009 3:32:28 PM] System.Runtime.InteropServices.COMException
- The directory name is invalid. (Exception from HRESULT: 0x8007010B)
- Source: System.Deployment
- Stack trace:
at System.Deployment.Application.NativeMethods.CorLaunchApplication(UInt32 hostType, String applicationFullName, Int32 manifestPathsCount, String[] manifestPaths, Int32 activationDataCount, String[] activationData, PROCESS_INFORMATION processInformation)
at System.Deployment.Application.ComponentStore.ActivateApplication(DefinitionAppId appId, String activationParameter, Boolean useActivationParameter)
at System.Deployment.Application.SubscriptionStore.ActivateApplication(DefinitionAppId appId, String activationParameter, Boolean useActivationParameter)
at System.Deployment.Application.ApplicationActivator.Activate(DefinitionAppId appId, AssemblyManifest appManifest, String activationParameter, Boolean useActivationParameter)
at System.Deployment.Application.ApplicationActivator.PerformDeploymentActivation(Uri activationUri, Boolean isShortcut, String textualSubId, String deploymentProviderUrlFromExtension, BrowserSettings browserSettings, String& errorPageUrl)
at System.Deployment.Application.ApplicationActivator.ActivateDeploymentWorker(Object state)

COMPONENT STORE TRANSACTION DETAILS
* Transaction at [10/15/2009 3:32:28 PM]
+ System.Deployment.Internal.Isolation.StoreOperationSetDeploymentMetadata
- Status: Set
- HRESULT: 0x0
+ System.Deployment.Internal.Isolation.StoreTransactionOperationType (27)
- HRESULT: 0x0

Thanks to Russell Christopher for blogging the only solution I have found that works: delete everything in the Apps folder. This should be safe given that ClickOnce applications repair themselves if completely removed. Apart from Start Menu shortcuts, they are completely defined by the files on the disk (I think).

Obviously this is the clients fault for messing with the files in the first place, but I was surprised it didn't repair itself.

Plain Text / XML .application File

Pathology: When the .application file is clicked, the browser shows the plain text XML file instead of launching the ClickOnce installer.

I only get this problem with IE6, and in my case it is a MIME issue. My server labels .application files with the generic application/octet-stream MIME types, and IE6 doesn't do a good job at deducing the correct type by inspecting the content.

It appears that IE7 and above have more sophisticated ways of resolving the .application file MIME than simply looking at the MIME type that the server reports, which is often difficult or impossible to change for exotic types like application/x-ms-application if you don't have root access to the server. In this case, you can serve up the EXE bootstrapper instead of the .application file to anyone with IE6.

Clue 1: Probably a server MIME encoding issue.
Clue 2: More evidence of MIME encoding.

The ClickOnce Graveyard

Here lies all the problems I am currently experiencing for which I do not have a definitive solution. As they are resolved, I will move them up into the body of the main article.

Setup EXE Bootstrapper Fails to Launch

Pathology: On a virgin XP installation, the EXE bootstrapper fails to launch.

Clues: Saving the bootstrapper and launching it manually throws a dialog with the following example text:

Unable to satisfy all prerequisites for [Application]. Setup cannot continue until all system components have been successfully installed.

Prerequisite check for system component .NET Framework 3.5 SP1 failed with the following error message:
"Installation of the .NET Framework 3.5 SP1 requires Windows XP SP2, Windows 2003 SP1, Windows Vista, or later. Contact your application vendor."

See the setup log file located at
[Log file location]

This doesn't happen on XP SP2 or above, but I don't know how to make this a prerequisite. More to the point, I don't know how to throw a friendly dialog if this prerequisite is not met.

Sinister discovery of the day

2009-10-01T10:29:00.003+01:00

Whilst researching international post-codes (don't ask), I happened upon a Wiki page about Chinese post-codes (not a euphemism). Note the codes (000000–009999) reserved for Taiwan.

Apology for Alan Turing

2009-09-11T12:21:00.004+01:00

The British Government apologised for their persecution of Alan Turning following an online petition. I'm delighted by this for both highlighting the abominable treatment of gay people and for bringing my hero Turing to peoples attention again.

As a signatory of the online petition, I received a copy of the apology in full in an email (which I thought was a nice touch). Here it is reproduced in full:

"Thank you for signing this petition. The Prime Minister has written a response. Please read below.

Prime Minister: 2009 has been a year of deep reflection – a chance for Britain, as a nation, to commemorate the profound debts we owe to those who came before. A unique combination of anniversaries and events have stirred in us that sense of pride and gratitude which characterise the British experience. Earlier this year I stood with Presidents Sarkozy and Obama to honour the service and the sacrifice of the heroes who stormed the beaches of Normandy 65 years ago. And just last week, we marked the 70 years which have passed since the British government declared its willingness to take up arms against Fascism and declared the outbreak of World War Two. So I am both pleased and proud that, thanks to a coalition of computer scientists, historians and LGBT activists, we have this year a chance to mark and celebrate another contribution to Britain’s fight against the darkness of dictatorship; that of code-breaker Alan Turing.

Turing was a quite brilliant mathematician, most famous for his work on breaking the German Enigma codes. It is no exaggeration to say that, without his outstanding contribution, the history of World War Two could well have been very different. He truly was one of those individuals we can point to whose unique contribution helped to turn the tide of war. The debt of gratitude he is owed makes it all the more horrifying, therefore, that he was treated so inhumanely. In 1952, he was convicted of ‘gross indecency’ – in effect, tried for being gay. His sentence – and he was faced with the miserable choice of this or prison - was chemical castration by a series of injections of female hormones. He took his own life just two years later.

Thousands of people have come together to demand justice for Alan Turing and recognition of the appalling way he was treated. While Turing was dealt with under the law of the time and we can't put the clock back, his treatment was of course utterly unfair and I am pleased to have the chance to say how deeply sorry I and we all are for what happened to him. Alan and the many thousands of other gay men who were convicted as he was convicted under homophobic laws were treated terribly. Over the years millions more lived in fear of conviction.

I am proud that those days are gone and that in the last 12 years this government has done so much to make life fairer and more equal for our LGBT community. This recognition of Alan’s status as one of Britain’s most famous victims of homophobia is another step towards equality and long overdue.

But even more than that, Alan deserves recognition for his contribution to humankind. For those of us born after 1945, into a Europe which is united, democratic and at peace, it is hard to imagine that our continent was once the theatre of mankind’s darkest hour. It is difficult to believe that in living memory, people could become so consumed by hate – by anti-Semitism, by homophobia, by xenophobia and other murderous prejudices – that the gas chambers and crematoria became a piece of the European landscape as surely as the galleries and universities and concert halls which had marked out the European civilisation for hundreds of years. It is thanks to men and women who were totally committed to fighting fascism, people like Alan Turing, that the horrors of the Holocaust and of total war are part of Europe’s history and not Europe’s present.

So on behalf of the British government, and all those who live freely thanks to Alan’s work I am very proud to say: we’re sorry, you deserved so much better.

Gordon Brown

If you would like to help preserve Alan Turing's memory for future generations, please donate here: http://www.bletchleypark.org.uk/

Petition information - http://petitions.number10.gov.uk/turing/"

I won't link to the Downing Street site version because the comments might as well have come from the Daily Mail's letters page.

BBC News: File-sharers' TV tastes revealed

2009-08-28T12:13:00.002+01:00

The BBC have a story based on some download statistics for TV & film that makes interesting reading. If the TV network executives read this and think "threat" rather than "opportunity" they are missing a trick. What if they uploaded the torrent themselves simultaneously with transmission including the adverts? Nobody is going to bother with an illegal version if this one is available immediately and is of high quality. Only a hardcore pirate would bother with an ad-free version because this would lag slightly behind the official version in terms of time to release, but crucially the number of seeds, and hence the download speed, would be significantly lower.

Yes people could zap through the ads, but that is no different to "TIVO" on broadcast anyway and only a schmuck watches things live these days anyway. If only 2% of people sat through the ads, that is still 1 million people you don't have today in a demographic you would normally struggle to reach.

There is a secret power in "global" as well. Advertisers talk about "playground repeats", which is when slogans and jingles are repeated in the school yard. If your adverts start to permeate the global culture a synergy emerges where the phrases and concepts become a lingua franca in online conversations.

Infrastructure requirements: a laptop, a big disk, and a decent uplink; not bad if you want to reach more than 50 million people worldwide. If it really takes off you could even do away with that expensive array of radio towers.

A halfway solution is no good. Custom players, region and DRM locking, proprietary standards, etc... will always be a niche play against the power of free. Open it up and let the community build the software and hardware to support it. Stick to what you know: making TV shows.

I don't pretend that this is a good solution for films, but it is perfect for TV.

ASP.NET File Permissions

2009-08-21T12:06:00.003+01:00

Short Story:

Use Process Monitor to help resolve ASP.NET file permission problems.

Long Story:

I have some files on one machine that I would like to make available through one of our company websites. We use IIS for our hosting, so I created a file share on the machine with the files and linked to it using a Virtual Directory. Now when resources are accessed via IIS they do so as the IUSR account (unless other authentication options are enabled), so I added read permissions for this user to my files security settings and to the sharing permissions (I always forget to do both), but it did not work. There may be some fancy ASP logging that tells you why, but I have always struggled to get the ASP logs to do anything very useful. Several smart people on the Internet recommended using Process Monitor from Microsoft (previously from Sysinternals). Everything that the Sysinternals guys make is made of pure awesome, and this is no exception. It allows you to search for the ACCESS DENIED operation (a CreateFile function call), and this shows all of the relevant details, including the user credentials used. In my case it was IUSR, or actually /IUSR_, which is some sort of domain equivalent. Exactly the user I had set the permissions for, so why wasn't it working?

Running Process Monitor on the file hosting machine instead reveals the problem: the file access was impersonating a different user, namely /$. It seems that this is the default behaviour across machine boundaries unless you are explicitly impersonating a designated user, in which case those credentials are passed intact.

The fix I chose may not be the recommended solution, but it worked for me: explictly impersonate IUSR to force the credentials to be passed across the machine boundary. To do this you need the IUSR password, which can be obtained by following this helpful advice. Whatever you do, don't change the IUSR password or you may unwittingly open a portal to a new dimension of pain.

Richard Feynman: The Character of Physical Law

2009-07-20T08:57:00.002+01:00

Feynman's incredible lecture series is now freely available on video here: http://research.microsoft.com/apps/tools/tuva/index.html.

It is wonderfully complete, including the VT header, which I think should begin all videos.

It's all made possible by someone called "Gates".

eMail for Kids

2009-05-25T10:04:00.002+01:00

#1 son is of an age where he would love an email address. There are many pay for services that offer safe kids emails, but it seems like a dumb thing to have to pay for.

The must have feature is an exclusive whitelist. This means that ONLY people on your contact list can send you emails.

My beloved GMail don't offer this service, but Microsoft's Live Mail does. I will let you know how it works out. It took a while to make sure all the permissions were locked down.

Incidentally, the default account name for my location is live.co.uk or hotmail.co.uk, which is lame, so I experimented with the URL and found I could trick it into US mode like this: https://signup.live.com/signup.aspx?mkt=EN-US&rollrs=12&lic=1. This gives you the option of registering for live.com or hotmail.com addresses.

Visual Studio 2008 Product Comparison

2009-03-30T10:22:00.002+01:00

The Visual Studio product page appears to have no comparison chart that spans all product editions. This is quite an omission for a product line that costs anywhere from $0 (for the Express editions) to $10939 (for Team Suite with an MSDN subscription). A range I would also describe as going from "outrageously amazing" to "poking developers in the eyes with toothpicks". If you dig around for a while, there is at least a price comparison chart on the site. However, you have to venture further afield for a clear feature comparison chart on the general Microsoft download site.

One price step that interests me is between Professional and Team System Developer. These are likely to be the two editions most commonly compared by price conscious code shops. If you use Team Foundation Server, you will need to add a Client Access License to Professional, which takes the total to $800 + $500 = $1300. Feature wise Developer edition gives you:

Code Profiling
Code Analysis (including check-in policies and spell checking)
Code Coverage

With Developer you are also required to buy MSDN Premium, so the total cost is $5469. Professional + CAL + MSDN Premium is $2500. So you have to ask yourself if MSDN Premium is worth it, and if the extra Developer features are then worth doubling your investment.

101 LINQ Samples

2009-03-24T17:00:00.003+00:00

(via MSDN)

Restriction Operators

Projection Operators

Partitioning Operators

Ordering Operators

Grouping Operators

Set Operators

Conversion Operators

Element Operators

Generation Operators

Quantifiers

Aggregate Operators

Miscellaneous Operators

Custom Sequence Operators

Combine

Query Execution

Virtual Earth Silverlight Map Control CTP

2009-03-21T13:59:00.003+00:00

The long anticipated Silverlight control for Virtual Earth took a step closer yesterday with the release of a very mature CTP. The experience is nearly identical to the old HTML map (which they call the AJAX map), but with the smoothness of Silverlight's DeepZoom. They also have an interactive SDK, just like the old one, with clear migration hints.

The really exciting thing about it is the fact that you can render any aribtrary Silverlight XAML on top of the map. This opens up Google Earth quality content in a browser, which raises the stakes yet again in the GIS war. Add this to the location awareness zeitgeist and we have some interesting times ahead.

I'm still confused about how they plan to maintain both the Silverlight and VE3D platforms. Is the goal ultimately a single platform? Given that one is based on managed DirectX and one is based on the Silverlight XAML subset (which doesn't do 3D), I can see no technical middle ground. Perhaps they will harmonize the SDK's to make it easy to write for both, but not knowing which to bet on makes me uncomfortable.

Interactive SDK (you probably need to register with Microsoft Connect)

Chris Pendleton @ MIX with the nerds wot wrote it. Some nice examples of the free-form XAML layering.

Domain	Clicks	Impressions	CTR
radiolluvia.com	2	1	200.00%
net-ebooks.com	1	1	100.00%
umtsfree.net	1	1	100.00%
littleabout.com	1	1	100.00%
mtncareer.com	1	1	100.00%
jonefm.com	1	1	100.00%
radiobendele.com	1	1	100.00%
iphalloween.info	1	1	100.00%
pdfee.com	1	1	100.00%
rf-online.com	1	1	100.00%