Jason Cartwright

getFavicon

I've been using Google's favicon fetching API to illustrate links on the play.tm wire and jasoncartwright.com for a while now, and it's been working fairly well. When given a domain the Google system returns a PNG of the domain's icon. The problem was that Google's system appears to rely on the favicon being at example.com/favicon.ico, which isn't always the case.

Some sites, for whatever reason, choose to place the favicon somewhere different and reference it from a tag in the of their pages. To find the URL of the favicon you therefore have to (expensively) download the entire HTML page, and parse it.

Frustrated by the limitations of the Google API I decided to solve the problem using Google App Engine, by creating getFavicon.

The first problem I came across was when just requesting the /favicon.ico file. Lots of people having poorly configured web servers that return a variety of exotic HTTP response codes, or even a 200 and zero byte body. It's quite annoying to deal with all these exceptions.

To retrieve a favicon specified in the site's HTML I've used the superb BeautifulSoup to do this. This library hugely simplifies the parsing of HTML, but still doesn't completely solve finding the correct URL of the favicon. The value varies greatly - "shortcut", "icon", "shortcut icon" and "favicon" are all values I've seen.

Of course, all this page & icon downloading and parsing is pretty heavy, so a two-tier cache is used - memcache for a rapid, temporary store and AppEngine's datastore for a slower, persistent store. It works well, and keep the app ticking over quickly. Well over 90% of the icons served come from a cache.

webmytop - MySQL monitoring tool

A few years ago I made an ASP.net version of Jeremy Zawodny's mytop tool, to use monitoring the MySQL servers that power play.tm.

The tool consists of two simple parts - some general uptime, number of query, etc stats and the process list MySQL generates with the query "show processlist".

The idea was to make it web-based and have the stats updating in near-realtime using AJAX. It worked a treat and many a poorly coded query has been reoptimised after this tool shows it up churning a machine.

Things have moved on since - now my language of choice is PHP and AJAX is all done through jQuery, so it was time for an update. I also figured that other people may be interested in using it, so we opensourced it.

You can find the source and download packages over here at code.google.com/p/webmytop.

Simple PHP execution time monitoring

We've all heard about optimising the user's experience on the client-side, but what about on the server?

When building out play.tm we decided we needed a tool that broke down how the PHP script generated the page and displayed how long each element took to generate. We could then optimise the worst performing code. We called it 'speedtrap'. So, here is the code:

In the top of your page:


$st_StartTime = microtime(true);
$speedTraps = array();

The function:


function speedTrap($trapName) {
	global $st_StartTime,$speedTraps;
	$difference = microtime(true) - $st_StartTime;
	$speedTraps[$trapName]=$difference;
}

At every point where you want put a speedtrap, with a description of what has happened in the code:


speedTrap("Done some part of the page");

Then, at the end of the page call this function:


function speedTrapBars() {
	
	global $siteVars,$speedTraps;
	
	if ($siteVars['showSpeedTrap']) {
	
		?>
		SpeedTrap!
		
		 $value) {
			?>
				
				ms
			
		Server Done
		ms
		";
	
	}
	
}

But wait! There's more! Some bonus CSS:


#speedtrap {
	font-size:1.1em;
	margin:5px;
}
#speedtrap dt, #speedtrap dd {
	float:left;
	margin-bottom:1px;
}
#speedtrap dt {
	width:100px;
	clear:left;
}
#speedtrap dd {
	background:lightgray;
	padding:2px;
}
#speedtrap .last {
	background:gray;
}

Blog Redesign

After several years with a questionable design aesthetic and some dodgy markup my blog has remerged with a fresh look, more content and some technical optimisations.

The most striking change is the activity feed on the left of the homepage. I've been playing with lifestreaming services for a while and none of the aggregators I've used have cut it. Then came FriendFeed. The combination of quick updates, a high number of supported services, large user community (including many of my friends) and arguably most importantly a quick and simple API makes using this service a no brainer. This simple feed is processed by an XSLT script and then displayed (with some help from CSS, and jQuery - more on that later). The XSLT is pretty straightforward and is available for download. You'll probably want to customise it for your needs, but drop me a line if you do use it and make the source available, if possible.

There are a couple of tools I've used to augment the lifestream. First is the favicon. This is pulled in using a Google service that I probably shouldn't be hotlinking. The XSLT puts an image in the HTML pointing to http://www.google.com/s2/favicons?domain=[domainname], for example. This returns the favicon for the domain in question (yes, I know the "Content-Type" header is wrong - maybe we'll see that fixed at some point). Should the system know that a favicon doesn't exist it will display a generic earth icon. Very useful. Perhaps I can get that header bug fixed at some point - along with maybe getting it sent as a GIF or PNG rather than a JPG... a poor choice I'd say, but I'm sure there is a reason.

The mouseover functionality is provided by my javascript library of choice: jQuery. Not much to say here, it's really good and really fast. The twist is that it is hosted centrally at Google. This new service (as of a few days ago) provides a standard URL for the version of jQuery I'm interested in, which is really cool because that URL may (with increasing probability, as more webmasters catch on) already be cached on the user's machine therefore saving the 17kb or so hit the library costs.

I've not had comments on my blog before, mainly to save the time of moderating them or dealing with spam. In the past some people have criticised me for harping on about Web 2.0 technologies then not having the basic visitor interaction of blog comments on my own site. So, time to fix this and the solution came in the form of disqus - a hosted comment system that when you place little widgets on pages allows users to comment on those URLs. Initially I was wary of putting someone else's javascript on the page, forcing a system on my blog readers, and giving the ownership of the comment text to a third party, but disqus did a good job of winning me over. The key paragraph was in this help page:

Who owns the comments? Short answer: You do.

I've seen occasional 500 Internal Server Errors from disqus, but other than that they are rock solid. A decent API gives you access to the comments made too - which is just the ticket should I decide to move away from it.

Not much has changed on the backend of the blog, it's still written in ASP.net which was the quickest solution for the Windows boxes the site is hosted on. ASP.net's solid caching functionality keeps everything ticking over, not stressing the MySQL boxes that hold the data or hitting web APIs too often.

One other small point is that I no longer obfuscate the HTML and CSS behind the blog. As a web developer I probably get more people peeping at the source code than is usual, to see what I'm up to. Minimalistic and less descriptive client-side code makes the source less readable and therefore understandable but the benefit is a decrease in transmission size. However the benefit is so small on a tiny site like this when the content is gzipped its not worth losing the readability over.

So, come back in a few months when I get bored of this design and code!

MADDness

Article first appeared here, and was edited by Luke.

The high profile release of Grand Theft Auto IV this week has seen a number of groups use the inevitable publicity for their own means. Newspapers, online sites (yes, including this one) and blogs eager for pageview-generating stories about the hit title lap up stories from pressure groups, hoping to ride in on the search results or the Digg/Slashdot/Reddit homepages the masses consume. This potent and symbiotic relationship is well documented, and at no time is it more visible than during a big event - and GTA IV will probably be the biggest we'll see this year.

Not the least publicity hungry of these parties is, of course, our old friend Mr Jack Thompson. He waded in with a typically unmeasured rant calling the 18/Mature (depending on jurisdiction) rated game "the gravest assault upon children in this country [the USA] since polio". So far, so predictable.

The inclusion of a drink driving element to GTA IV also brought it to the attention of an organisation called "Mothers Against Drink Driving" who promptly put out a statement saying "MADD is calling on the Entertainment Software Ratings Board to reclassify Grand Theft Auto IV as an Adults Only game, a step up from the current rating of Mature and for the manufacturer to consider a stop in distribution - if not out of responsibility to society then out of respect for the millions of victims/survivors of drunk driving."

This 'feature' sees the main character patronise a bar, then the game allows you to drive afterwards. Driving whilst in this state is inevitably difficult with the car becomeing understandably very difficult to control. The police also take an interest and give chase when they see your drunken antics. All quite realistic then, and probably not a good thing to be doing as it doesn't even aid the main aim of the game - to work your way up the criminal career ladder.

So who is this pressure group? Mothers Against Drink Driving (MADD) was set up in the 1980's by Candice Lightner with the laudable aim of reducing drink driving in the United States, a country that has a one of the worse drunk driving accident rates in the developed world. After forming, the organisation rapidly turned prohibitionist, and began advocating less liberal policies tackling issues with alcohol in general rather than just drink driving, such as raising the drinking age. Civil liberty groups also criticised the organisation for advocating alcohol interlock devices factory-installed in all new cars. It has also been pointed out that there is a potential conflict of interests between MADD and some of it's bottle industry partners, causing MADD to refrain from advocating increased taxes on distilled spirits.

Lightner left the group in 1985 saying that it "has become far more neo-prohibitionist than I had ever wanted or envisioned. I didn't start MADD to deal with alcohol. I started MADD to deal with the issue of drunk driving".

Thompson often calls GTA a "murder simulator", so it is ironic to hear that MADD, critical of the drink driving depicted in the game, themselves ran a drink driving "simulator". Started in 1988, in partnership with Chrysler and the US Department of Transport, MADD rigged a Dodge Neon car to simulate driving whilst drunk. A computer took the driver's weight and a hypothetical number of drinks, then delayed the steering and braking accordingly. Naturally some safety measures were employed like a trained driver with a brake pedal in the passenger seat, but what they created was effectively a full-size real-life drink driving simulator. This was sent around hundreds of US schools and college campuses and driven by hundreds of thousands of students.

So, it's acceptable to drive a deliberately impaired full-size car around but not alright to simulate the same thing on a game console? The hypocrisy here is troubling, especially when its digested so fervently by media with their own interests to serve. Whilst nobody is arguing that the group's core aim is not admirable, how can we take them seriously if over-zealous statements like this are made, especially when MADD have already set their own precedent when it comes to acts of 'simulation'. Your comments, as ever, are appreciated.

What Apple should have done, by ASUS

Article first appeared here, and was edited by Luke.

With all the hype surrounding the Macworld Expo and coverage Stevie's keynote has produced, you could be forgiven for believing that the Macbook Air is the be-all and end-all of the petite laptop market.

Sure, its going to be a solid product with all the traits of expert engineering that makes Apple systems highly buyable even when sold at their higher-than-norm premium price point. The MagSafe power cable, the magnetic screen hinge, the solid hardware-software integration are all going to be there. However, the key selling point appears to just be a kudos factor that probably won't affect the average consumer - the thinness. Whilst it might be an excellent bragging right when it comes to your mates in the office, I fail to see how this translates to making the Macbook Air a fundamentally better product. It will take up marginally less room in your bag (and you won't need those Firewire devices or ethernet cable due to lack of ports) and weigh less, but will it make it more usable? Probably not - it still won't fit properly on an aircraft tray table.

This knowledge and the intense reality distortion field a Stevenote produces have hyped the Macbook Air - but we're already beginning to see the backlash from differing opinions. Aside from the obvious speed hit Apple have taken by using the tiny iPod-style 1.8" disk (although the much faster SSD is available for $999 more), and the sparsity of ports available, the main problem I see is still the size.

The solution should have been to make the form factor smaller, not just take the Macbook to Weight Watchers. This would mean that you could fit pretty much all the standard ports onto the product and still make it lighter. With all the dimensions practically the same as a normal laptop, can it justify the lack of functionality the thinness apparently necessitates.

One company has done this, and it's all looking rather good. ASUS, better known for making motherboards, has produced the "Eee". Now, I said the "Wii" was a stupid name, and I've not lived it down since Nintendo hit gold with that product. I fear I'm going to make the same mistake here - Eee is a ridiculous name.

This subnotebook is pretty darned small. The 7" screen is in a 225 × 165 × 350mm case and weighs just 0.92kg. Lets compare this to the Macbook Air with a 13.3" screen sitting in a 227 x 324 x 194mm case, weighing 1.36Kg. The main problem with this tiny screen is that it can only support upt o 800x480 resolution, and this falls short of Windows XP's 800x600 minimum stated requirement. This is not to say that the machine can't run XP, it can, but with the incredibly annoying screen scrolling.

Processing juice comes via a 900Mhz Intel Celeron chip (clocked to 800mhz on the lowest spec model), with upto 1Gb of RAM. Storage is provided by 4 or 8Gb solid state disks. You'll probably have more storage on your iPod, unfortunately. Connectivity is via all the usual suspects. Somehow ASUS have managed to cram 3 USB ports onto this sucker, alongside a MMC/SD card reader, 10/100 ethernet, mic and headphone jacks and a VGA port. 802.11b/g also features, and is probably going to be the main connection method you'll use, to take advantage of the small device's size.

The solution to the screen scrolling problem is to bundle the hardware with a version of Linux (Xandros) with rather useful versions of OpenOffice, Firefox, Skype, and links to freebie online services such as Wikipedia and Google Docs. This is all tied together with a clean, custom, tab-based UI. All very impressive, and excellent for the average home user. An enthusiast-class user will however have it all wiped as soon as the machine is unpacked and their favorite flavor of Linux installed in a jiffy without, it should be noted, paying the Windows 'tax'.

Speaking of money - did I mention that these beauties cost from as little as 215 GBP or 300 USD? Its not a fair comparison, but the Macbook Air starts at 1200 GBP or 1800 USD. Little wonder ASUS have shipped 300,000 in the last year.

The big news however is that ASUS plan to sell several million units in 2008, by making an updated versions of this device. In what must be a design and manufacturing nightmare they are going to be producing 7, 8 and 8.9" models the biggest of which will apparently feature a touch screen. The fan from the older range will be removed, due to the use of more efficient Merom-class Intel processors, resulting in a 36% drop in power consumption, and we can expect a hike in SSD size. A version of the machine with WiMax also demoed recently. Presumably ASUS will have noticed the Windows screen resolution problem and will take action on that front.

Expect these puppies out in April, beware Apple.

Content Findability: keyword, not URL

In making a purchasing decision a buyer will go through a process often expressed as "AIUAPR". This acronym stands for the phases a buyer almost always moves through before laying out the cash...

Awareness
Interest
Understanding
Attitudes
Purchase
Repeat Purchase

Despite the rise of the internet and all the hype surrounding Google etc there is still no advertising medium quite as good at raising awareness for the average consumer as TV. Other advertising mediums such as radio, outdoor and ambient media still have their niches in ad buyer's media mixes, however along with TV all these often fail at the "Interest" stage and have almost definitely finished being consumed before the potential buyer has hit "Understanding". This is why infomercials and home shopping channels whose revenue depends on people handing over the cash there and then spend so long describing each apparently simple (and usually crap) product.

Therefore the best way to ensure your "Awareness" and "Interest" spend is more likely to be converted into a sale, by moving the buyer down this decision chain, is to provide more information than you can fit in the average TV/radio/outdoor advert. A website of course can fulfill this task well by conveying a lot more information about a product than these traditional adverts ever can - often presenting such information in a more persuasive form with interactivity.

The problem therefore is getting your potential customer to traverse from offline to online - the advertising time is expensive and URLs you have to convey to users are complicated beasts...

h-t-t-p-colon-forwardslash-forwardslash-w-w-w-dot-your-brand-dot-co-uk

Marketeers, rightly, become particularly hung up on conveying their URL if their product is purchasable at the website or, even worse, the URL is the product. This usually results in inane adverts where the URL is repeatedly barked at the user, and inevitably people take the piss on YouTube...

There are ways to optimise the URL, of course - such as reducing the length by not mentioning the "http://" and/or "www.", or use of mobile shortcodes to request information. The use of branding slogans as URLs is also prevalent - a practice I always think is quite shortsighted and appears to usually be the advertising agency getting wrapped up in the campaign not seeing the bigger picture of their customer's brand.

Recently I've been seeing everyone from criminals to the government giving prompts to use search engines with specific keywords, rather than just giving a URL, in order to get their audience to more in-depth content that supports the agenda they are pushing. Just as companies use the limited information put in adverts to seed interest in a product, this new breed of advert is aiming to provide a simple pointer further their influence by getting people to seek out more data.

Here are some examples...

Act on CO2

The UK government's "Act on CO2" campaign spans several mediums - most notably quite a few high-profile TV spots. Note at the end, the call to action is "Search online for 'Act on CO2'" rather than a URL.

Grassroots Protests

"Building 7" is a reference to 7 World Trade Center, a building that was destroyed in the terrorist attacks on 11th September 2001. Conspiracy theorists argue that the building was reduced to rubble in a controlled demolition, then the media covered this fact up. Whatever you think of these theories - the use of 'google' as a verb is interesting as it provides a simple shortcut to show that the information is available online, and it's many locations (over 67m search results at the time of writing - with wtc7.net, the top one).

In this example, a protest auction on eBay, the person pushing information has used a reference to web search over a URL, or even a link. This may be because their HTML skills aren't very good - but the result is arguably just as persuasive.

Here we can see a Scientology protester holding a sign promoting anti-Scientologist content. Apologies for the quality of the video capture - I can't read it either

So what is the upshot of this? The first of the obvious risk of the user not making it to the content you expected them to. A Googlebombing (or just some shrewd SEO) by someone with a different agenda could modify the search results. This is arguably quite difficult as the original website will probably be linked to more often by higher quality sources than an alternative or opposing website - as they have the head start.

One way people can hijack your advertising is the use of sponsored links. On Google in the UK we can see this, where people advertising carbon offsetting and wind turbines have advertised. Interestingly, the last link is some chancer with the completely unrelated tyre puncture protection product. Obviously the people who have shelled out for the TV adverts have more financial clout than your average sponsored link buyer - so they can easily outbid anyone with different motives.

It's impressive to see advertisers of the legitimate and (arguably) illegitimately using this technique to move buyers down the purchase decision chain - after all, it is how users find content organically on the net.

New BBC Homepage

The BBC have launched their new homepage that has been in beta for the last few months - it looks and feels great. To get this out the door through the restrictive BBC standards and ancient infrastructure is one hell of an achievement. Good work.

However, the customisability of the page is completely knobbled by the lack of content and some basic persistence problems. The page lacks fundamental functionality that even the most basic Pageflakes-a-like customisable homepage has baked in.

For starters, the layout isn't saved in anything other than your cookie so I have to customise the page on every machine I use. This persistence problem gets even worse when you try and use other BBC systems. When I enter my postcode to customise the homepage, this information doesn't appear to be used to modify the content on any other site, and I still get asked for my postcode or place name elsewhere - /weather, /whereilive, and the customisable news homepage (a competing homepage... on the same site - only at the BBC!) for example.

The other problem is the lack of third-party content. Again, pretty much every other customisable homepage on the web allows you to develop widgets/gadgets/whatever that are dumped on the page in the positions you require. Gadgets are even going social now - with open specs like OpenSocial actively encouraging widget reuse across the web. Not so at the BBC, no external gadgets are allowed (not even basic external RSS feeds), and you can't repurpose the "gadgets" they use for use elsewhere.

Some of the content on the page seems a little pointless - I can't add stock quotes in the money section (even though the BBC has the content on the news site), and some blocks just consist of a link. The BBC's flagship iPlayer content is just a linked image to /iplayer - not very persuasive.

The upshot is that the design is great and it gives one of the UK's biggest sites a fresh feel, but all the space (and download time?) used by the customisation controls is wasted because the user can't make the page their own, like they are used to elsewhere. Don't even think about adding Sky's news, your mate's blogs, or anything more useful. Here is our news feed... Auntie knows best.

8 Facts About Me Meme

I don't usually go in for these chain letter, MySpace style questions about yourself things, but as I've been tagged in one by non-other than the big man himself Mr Forrester, I'll do this one. He called me "a funny geezer" too.

Here are the rules...

Each player starts with eight random facts/habits about themselves.

People who are tagged need to write a post on their own blog (about their eight things) and post these rules.

At the end of your blog, you need to choose eight people to get tagged and list their names.

Don’t forget to leave them a comment telling them they’re tagged, and to read your blog.

I was born on Friday 13th. This is the standard 'fact about myself' I roll out in those team building things - I'm sure more a few people have heard me say this one several times. It's usually accompanied by a chorus of "well, that figures" afterwards.

My grandma has more IT qualifications than I do. People I work with find this one a bit strange - I have no A-Levels or degree. I got my break at a dotcom in 1999, thanks to two reprobates called Ross and Kevin. Getting a gig at Google was difficult without a degree, but thankfully they gave me a chance too. My grandma has a City & Guilds qualification in some practical IT skills - good on her.

I'm from Devon. Its a great place, as long as you're not young. I got out just in time to save my sanity, as the only way to make any money is to run a hotel or cater for tourists in some way. Perhaps I'll retire there. I'm also half-Cornish, and was christened there.

My dad makes ice-cream for a living. Not strictly about me, but its a good'un. He makes italian style ice cream in Devon, which is a lot of fun but jolly hard work. I go and help him out as a holiday from tapping at a keyboard some summers.

I like Brixton. I moved to Brixton about 2 years ago, and I think its a great place. Despite the incredible amounts of bad publicity the place gets, it has quite a community feel and some fantastic places to eat and be entertained.

I can't drive. To people I know who don't live in London and don't know what an Oyster card is this is tantamount to social and employment suicide. I tried to learn, and got two majors on the test - squealing the tires on a left-hander, then forcing another car off the road made up the chap's mind. I'll learn again another time.

I enjoy talking about politics. Particularly US politics for some reason - if there was a version of Heat magazine that featured Gordon Brown and George Bush in the place of Amy Winehouse and Britney Spears (perhaps "Calefaction" magazine?) then I'd buy it. In fact, there is... Today in Parliament on Radio 4 and in podcast form.

I hate flying but love traveling. Flying with me involves watching me sweat like a nutter, hearing me let out little yelps, and feeling me crush your hand as I grip it like my life depends on it. This irrational fear is the bain of my enjoyment of every holiday that involves planes. Sat in the seat I constantly have to listen to the engine noise and feel if we're rising or falling in a ridiculous attempt to have some control over what is happening. This came to a head on a flight in a small plane in Malaysia where coming into land I actually believed I was going to die and strained a muscle tensing myself in the seat.

That's me done. Here are the next ones...

Luke
James - although I'm not sure how it will work with his one word shtick
Yuki
Seb
Andy

Gosh darned it, that's it.

eBay Items

I've gone and done a bit of early spring cleaning, and the result is a load of stuff on eBay.

Alternatively, checkout the rather cool widget...

iPlayer Disappointment

Surfing around some blogs I came across Adam Bowie's post about Charlie Brooker's Screenwipe. I'm a big Brooker fan, from back in the day when people bought these paper objects called magazines and Charlie wrote in PC Zone through to TV Go Home, Nathan Barley and now this BBC Four series.

Anyhow, Adam referenced the iPlayer URL of Brooker's end of year special of Screenwipe which I'd really like to see but missed due to being on holiday. I'd heard that the Flash version of iPlayer was released so I readied myself for a dose of Charlie's rantings to be streamed to my Mac and clicked the link. It wasn't to be. "Charlie Brooker's Screenwipe is no longer available - sorry!" the error message says. All very well - the rights aren't cleared for the BBC to stream this after 7 days, I understand that, but ultimately I'm very disappointed.

However, lets compare this experience to performing a search for the title shown in iPlayer: Charlie Brooker's Screenwipe Review of the Year.

1st result: Indirect YouTube link.
2nd & 3rd results: Direct YouTube links, and "More results from www.youtube.com" link
6th result: bbc.co.uk/programmes link, without iPlayer embed (because it wouldn't work, presumably) listing that I can view the programme on BBC Four at absurd times in the early morning.
7th result: torrent site

The "More results from www.youtube.com" link then gives me over 600 Screenwipe videos. The top 3 are the exact result I'm looking for (and they work - I eventually watched the show with my dinner from these URLs), but it also gives me last year's review of the year!

Whilst iPlayer has failed to be able to show me the programme that has been broadcast 8 days ago, the combination of YouTube and Bittorrent have been able to give me exactly what I was after instantly. It then allowed me to browse every Screenwipe programme that has ever been made, faster and in higher quality than iPlayer provides, and I could have done it whilst still on holiday. There are also user-editted clips, user comments, and user ratings.

BBC iPlayer: Making the unmissable, missable.

I'm in Malaysia

The tour of Hong Kong, Kuala Lumpur, Pangkor Laut, and Georgetown has begun. Have some photos below, or there are more over at Flickr.

XHTML Mobile sites are easy

Over at play.tm we've made a mobile site, at m.play.tm. This was made possible by a few different technologies and conventions all coming together. Google Adsense for Mobile for monentisation, XHTML Mobile for the markup, and the apparent de-facto standard of putting mobile sites on the subdomain m.[yourdomainname].

With all our content in the a database its no problem to output it to the simplified version of XHTML for use on mobile browsers, the design process was simple too - you can't display much on a tiny phone screen. Google Adsense adverts on non-mobile sites are embedded in the page using Javascript. The JS is executes and passes the page's URL to the script and Google's server-side code spits out the relevant ads (using Google's crazily complex algorithms to get to best ads, undoubtedly).

Mobiles can't usually execute Javascript, and the extra HTTP call creates added latency, which mobile pages are sensitive to. To get around this Google mobile ads are added to the page server-side, using whatever scripting language you choose. I guess Google figure that if you're savvy enough to create a mobile site then you are savvy enough to understand server-side scripting. Unfortunately Google only provide Perl, PHP, JSP and ASP v3 samples. As the main page generating parts of play.tm are written in ASP.net I created this function (based on the ASP v3 function Google provides), which of course you are welcome to copy and probably improve for your implementation (a mention and link would be nice though!):

function showGoogleAd(url)
	Dim googleTime, googleDt, googleScheme, googleHost,googlePubId
	
	googlePubId = "ca-mb-pub-xxxxxxxxxxxxxxxxxxxx"
	
	googleTime = DateDiff("s", "01/01/1970 00:00:00", Now())
	googleDt = (1000 * googleTime) + System.Math.Round(1000 * (Timer - Int(Timer)))
	googleScheme = "http://"
	if StrComp(Request.ServerVariables("HTTPS"), "on") = 0 Then googleScheme = "https://"
	googleHost = Server.URLEncode(googleScheme & url)
	
	Dim googleAdUrl, googleAdOutput
	googleAdUrl = "http://pagead2.googlesyndication.com/pagead/ads?ad_type=text&" & googlePubId & "&dt=" & googleDt & "&format=mobile_double" & "&host=" & googleHost & "&ip=" & Server.URLEncode(Request.ServerVariables("REMOTE_ADDR")) & "&markup=xhtml&oe=utf8&output=xhtml&ref=" & Server.URLEncode(Request.ServerVariables("HTTP_REFERER")) & "&url=" & googleHost & "&useragent=" & Server.URLEncode(Request.ServerVariables("HTTP_USER_AGENT"))
	
	Dim googleWebClient as New WebClient()
	googleAdOutput = googleWebClient.DownloadString(googleAdUrl)
	response.write (googleAdOutput)
	
end function

The upshot of this is that whilst it is still early days for building up our mobile readership and for Google's Adsense for Mobile Content programme we've found a new way of monetising our content. The upfront cost in developing this the mobile site was small as it just reuses a lot of existing code and infrastructure and it is incredibly simple in terms of design (the CSS file is just a few lines long) and content (simple markup).

Content delivery: insource or outsource

Its all gone a bit negative around here (abusing ITV, dismissing bbc.co.uk 2.0, and listing crap URLs), so I think I should write a bit about what I'm up to.

In the constant scaling battle that is running a large and growing content site a key decision is to choose where you put components of your systems. Do you insource, outsource or a mixture of the both?

So, on play.tm we've recently been moving around quite a few pieces of this jigsaw. To give a bit of background - we've got several million pages of content ranging from text to images to video, and we deliver this to around a million users a month. We're a small company where budgets are tight, but we operate in a competitive marketplace where expectation of the user experience are high. Most visitors have broadband, are experienced digital natives and we're competing with everyone from the world's largest media conglomerates to individual bloggers for their eyeballs and ultimately advertising revenue. To minimise budgets and maximise quality of user experience we are constantly striving to improve our technical infrastructure to punch above our weight.

I've talked before about Cachefly. We've been gradually moving play.tm CSS, Javascript and furniture images over to their CDN in order to speed up the site for users who may be far away from our London-based origin webfarm. The high cost of storage means that this works well for the small most hit upon files - we host just 5mb of files there, but it is hit on millions of times a day. However, it becomes too expensive to host content images (we have gigs of them) and videos (again, gigs of them and growing very fast). The only problem we've found with Cachefly is on some Asian connections the latency isn't much better than that to our London server farm. Anycast seemingly doesn't do its job properly and routes those users to San Jose, rather than the probably optimal Tokyo POP. On the whole though this service dramatically improves the speed (and scalability) of our site, particularly in off-beat locations, making us (anecdotally) faster than the competition whilst having an (arguably) more richly designed site.

Images and video are indeed a different game. Very large amounts of storage are required, making CDNs prohibitively expensive due the fact that they need to replicate your content over many POPs. Also, bandwidth on CDNs can be pricey. With next-gen consoles (well, the PS3 and Xbox 360) capable of outputting HD content, the screenshots and video from these games are increasingly of a very high resolution and therefore file size. Also, there is the cultural change that PR companies are becoming more aware of the internet as a medium to distribute video - and therefore are producing and distributing more of it. The upshot of this is a storage nightmare, what we need is inexpensive storage, bandwidth and most importantly the ability to increase the amount of storage available.

In comes S3, a web service from Amazon. Simple Storage Solution (SSS, or S3) allow you to store and transmit files with a simple model of paying for only what you use. So, we could purchase a big array of disks, the server(s) and software to make them spin, networking gear to make them communicate and the bandwidth for them to talk to the outside world. Problem here is that there is a huge upfront cost to buy all the storage we think we may need for years to come - we could get the planning wrong, and we'll probably only be utilitising a fraction of its capacity to begin with. S3 on the other hand allows us to pay for only what we use - GB storage per month, any bandwidth used, and a very small cost per HTTP transaction. Most importantly there is no limit on the amount of files we can store (if there is, it's ridiculously high), and as a bonus the service won't degrade if we get busy. This solution isn't the best for all usecases, but for our videos it works a treat.

One other area of the site we've outsourced is the RSS feed delivery. We found that RSS feed hits made up a very large proportion of the hits on the site - up to 50 hits/sec. Whilst we did our best to cache the script's output the capacity of the webheads could certainly be used for something more interesting than pushing this XML. The solution came in the form of Feedburner - this Google owned service takes a pre-existing feed and "burns" it, producing a copy that can be delivered to the end user. Since Google bought Feedburner they made their "Pro" services free, allowing access to lots of interesting analytics and to "burn" the feeds to a domain name of your choosing. As we already had many people using the feeds we opted to setup a new feed origin, let Feedburner get its content from there, then redirect all the old users to the burnt feed. It worked really well, and as a bonus Feedburner provide us with a load of detailed stats we previously didn't have access to. Feedburner's pinging service took another process off our boxes too.

When URLs go bad

We all know what makes a good URL. Its got to be hackable, meaningful, short, and most importantly its got to never, ever, change. But what happens when people get it wrong? Join me in a journey around the world of bad URLs...

http://www.ft.com/cms/s/0/4dc9d2ba-9f2e-11dc-8031-0000779fd2ac.html?nclick_check=1

CMS? Why do I care that a page is generated from a CMS? Does that identifier really need to be 36 characters long? Are the FT expecting to generate 36^36 (or more?) articles/pages in its lifetime? Just six characters (6^36 = 2.1bn) would probably cover it. You could push the boat out, go wild, and do seven (7^36 = 78.3bn).

http://prettymuchanyblog.com/archive/2007/12/31/the_blog_post_title/

This one has been hacking me off for a while. Many blogging systems put the permalinks to posts in an archive directory by default. To me, archiving implies something of historical interest that has been set aside from the main bulk of the posts. The URL could be used as an identifier of what we're actually expecting - a blog post, so why not make the link yourblog.com/post/2007/12/31/the_blog_post_title/ ?

http://iplayersupport.external.bbc.co.uk/cgi-bin/bbciplayer.cfg/php/enduser/std_alp.php
?p_cv=1.3&cat_lvl1=3&p_cats=3&p_new_search=1&p_sid=il25D1Si&p_accessibility=0&p_redirect=&p_lva=42

BBC URLs are usually of the beautiful form bbc.co.uk/[whatever], but this disgraceful effort breaks all the rules (I've had to break it in half to fit the page).

Stupid domain name - nobody cares where your server is!
Technology specific - cgi-bin, cfg, php (twice!), can you fit any more technology identifiers in there?
Ridiculous number of variables in the querystring, are they really all needed?

Why this URL can't be the following, we'll probably never know: http://www.bbc.co.uk/iplayer/support/finding_programmes/

http://www.livedepartureboards.co.uk/ldb/sumdep.aspx?T=DMK&S=VIC

National Rail Enquiries are renowned for having a crap website, even with Matt Somerville seemingly giving them free consultancy and showing them how to do it properly with the beautiful traintimes.org.uk. Checkout this URL porn: http://traintimes.org.uk/VIC/DMK, for the same page as above.

So, it seems what the National Rail people have tried to do is make a shortcut to get live departure information by registering and using livedepartureboards.co.uk. All very well, I know I like to check the train times before I walk out of the office into the rain. Problem is they then appear to have bodged it onto their nationarail.co.uk site. The result is repeating what site your on (the "ldp" directory - which I assume means Live Departure Boards) and a messed up interface. Users have been, rightly, drilled to realise that a domain name represents one site, which makes the "National Rail Enquiries" branding seem strange. Even worse clicking on "Home" takes you to the NRE homepage. The LDB homepage is a crap HTML redirect.

http://www-935.ibm.com/services/us/gbs/bus/html/bcs_financialmgmt.html?re=bcs_home
http://h71028.www7.hp.com/enterprise/cache/19054-0-0-225-121.html

Are the www-936 and h71028.www7 parts really necessary? It would be much better if these IT giants put these systems behind some clever load balancing system so that everything gets served from www.[companyname].com. Much less confusing, much more friendly.

bbc.co.uk 2.0: Why it isn't happening and shouldn't happen

Bit of background: I've recently quit the BBC after 3 years in the New Media dept (now called Future Media & Technology), and now work for Google. This is all my personal point of view.

Due to the unique way the BBC is funded it is rapidly becoming obsolete in the brave new world of the friction-free innovation that Web 2.0 affords. They are hanging in there, and respect for doing so, but for how long will the corporations online efforts be able to continue to do so?

The BBC's royal charter allows the BBC to exist, and was granted in 1927. It's been revised and re-granted every 10 years and says that the BBC "public purposes" are as follows:

sustaining citizenship and civil society;

promoting education and learning;

stimulating creativity and cultural excellence;

representing the UK, its nations, regions and communities;

bringing the UK to the world and the world to the UK;

in promoting its other purposes, helping to deliver to the public the benefit of emerging communications technologies and services and, in addition, taking a leading role in the switchover to digital television.

All this is wrapped up in the BBC's "educate, inform and entertain" remit, and is delivered to an undeniable and probably unique quality - just look at Radio 4's Today programme, bbc.co.uk/news and the children's offerings. I think we can summarise the BBC's reason for existence and funding via the charter with these assumptions:

TV & Radio costs a fortune to produce and distribute, therefore should be funded centrally
Good content has to be free from commercial and government control
New digital mediums need promoting
It's difficult to get content from abroad and to influence abroad
The long-tail of identities and communities can't be catered for in a mass-market commercial product

Whilst good TV and radio content still costs a lot of money to produce, distribution is now free or almost free. YouTube or the like, or P2P software replace expensive transmitter networks. Will we at some point see wholesale distribution of TV over IP? Probably, and radio to a lesser extent. Do new digital mediums need promoting anymore? Sky and Channel 4 are promoting HD and IP delivery of TV alright, and DAB is full of decent commercial stations now. Needless to say, its commercial interests that are driving innovation online. International influence is a doddle on the relatively borderless internet, and likewise longtails of small communities can be represented online to far greater extent than the charter envisages for the BBC's task.

The BBC's license fee funding is based on a broadcast model. The cost of producing and distributing the programme is fixed, and there is little or no variable costs that are acquired per user. A programme costs the same whether 10 or 10,000,000 people watch it. This 'push' broadcast model fits the BBC and its fairly fixed budget perfectly. Moving online we see a different model happening - a service here has both fixed and variable costs. First you need to develop the service in the first place (the fixed cost) then deliver it (the variable cost) - using servers and bandwidth, which have a cost-per-user. Whilst you can reduce the variable costs by driving the cost of each 'pull' transaction down by using cheap peering and the latest hardware and software - the cost is probably always going there (more on that later).

Commercial organisations (at least the good ones embracing the internet) find it easy to turn a profit from the variable costs of each user - it then just becomes a race to profitability by getting as many users as possible to cover your fixed costs.

One argument against this line of thought is that the tech will decrease in cost faster than the number of users will increase, therefore the BBC will just need to keep investing in new technology until the cost per user is so negligible it doesn't matter. Problem is that increasingly complex user expectations from online services mean that this will never happen. We're already seeing that the people using the technology the most efficiently (Google for example) are the most successful. The moment a new technique for decreasing the cost of these transactions (be it Moore's law, an equivalent such as Kryder's law, open source free software or a programming technique) a new cost comes along in the form of user expectation. Will this happen forever? Probably not, but we're a long way away from ubiquitous computing power and storage.

It's not just about computing juice though. Lets drag this blog post back to the BBC's problem with some examples. Flickr take my money, but it sure doesn't cost them $25 to serve my photos out for a year, which is probably the case for 99.9% of paying users. Posters to the BBC's 606 messageboard however did cost more in variable costs (technical and moderation costs) than the fixed cost that was allocated to the service. Dramatically so in fact, causing the site to have to be scaled back and restructured (much to the user's disgust) in order to stay alive. Upshot is that while a service like Flickr just needs to keep plugging away adding users to cover the cost of their employee's wages, the BBC has a substandard service and a load of pissed off licence fee payers.

This fixed vs variable income model brings about probably insurmountable cultural differences between successful internet businesses and the BBC as well. Auntie likes to have few, big, expensive, milestone projects to burn the cash in a predictable manner, whereas the more flexible internet industry takes a gamble on many small, inexpensive, iterative projects. "Please fail very quickly — so that you can try again" says Google's CEO Eric Schmidt. It's pretty much proven that a more agile, open and iterative approach produces services to a higher quality, and also to a release cycle length that the feature-clamouring user now rightly expects.

Moving away from the economic analysis of the situation facing the BBC, we can see the tide already turning. The BBC was an innovator in radio (2LO - in beta 1922, v1.0 when licenced in 1923) then TV (BBC Television Service - beta from 1929, v1.0 release 1946) but not now in the online age. Sky Anytime, 4OD, and ITV.com's video revamp have all launched before the BBC's iPlayer service (iMP beta 2005, iPlayer in beta, v1.0 not released at time of writing) showing commercial efforts in this field have trumped the BBC. One person working on the project called it "worse than boo.com". With the lead now lost, how can they pull it back?

Well, they have formed a Web 2.0 department to drive development of bbc.co.uk 2.0. This bizarre structure defeats the purpose of the sea-change they are trying to accomplish. Does Yahoo, Google, or any innovating startups have a Web 2.0 team? Of course not - the idea is ridiculous - they are Web 2.0.

Lets go through the some of the principals of Web 2.0 and the BBC...

Rich user experience: archaic BBC tech standards say that you can't rely on javascript/flash to deliver content, and pages need to be below 200kb in size. Buh bye innovative user interfaces, widgets/gadgets, Google or Yahoo Maps style interface, or YouTube for that matter.
User as contributor: BBC requires moderation of content before publishing it - see above for 606 example.
Participation, not publishing - as above.
Enable the long tail - BBC tech has limited ability to cater for large amount of content in the first place. CMSs are disparate and clunky, content distribution network is run off one single, overloaded computer (!).
Radical trust - this simply doesn't happen at the BBC, see 606. Not even to employee's, see first point.

All this seems very negative, and I suppose it is. The BBC is a big, easy target - but let me just point out the areas where I would defend the BBC. News and children's content is awesome, and with its non-commercial interest the BBC delivers the best services there are in these areas - even online. The problem I have is the BBC scrabbling to keep up in a trendy sub-sector of the industry where it's fundamental design makes it impossible to do so.

itv.com - what are they doing?

I went to itv.com the other day, not quite sure why, but I did. The first promo image was about Wayne Rooney's injury, which I was interested in so decided to give it a click...

That took me to the sport site (itv.com/Sport - capital letters in the URL, and linking to default.html for whatever reason) with same promo, so I click it again...

Then I'm on the football site (at itv-football.co.uk - not itv.com/sport/football, or itv-football.com) with yet another promo.

I'm getting hacked off at this point, but want to see it through, and thank god the click of this promo goes to the article...

Other annoyances include...

Some top nav item links open in a new window, and others don't
Some promos open in new windows, and other don't. This is particually annoying when it causes two video streams to play at the same time
Feedback form asks for your browser and OS - don't they know that already?
Most of the video doesn't work on a Mac. Why?
The dating site (not that I'm interested) is whitelabeled into ITV dating and FriendsReunited dating. Why bother?
Some content is needlessly written out using Javascript - presumably so that it can be updated centrally. Weird.

Flash Video Embedding

Yay! It works...

Cachefly Review

I first heard of Cachefly when they sponsored the diggnation podcast. I was impressed at the download speed of the popular podcast, and did a few tests on their network from various locations around the globe.

For those who don't understand what a CDN is here is a brief overview. Delivery of data on the internet is hindered by distance. The further your data travels to get to the person who requested it the more likely it is to be slowed down (called lag) or lost (called packet loss) by points in various networks that are overloaded or experiencing technical difficulties. There is also an inherent lag in the network caused by the sheer distance its travelling. For instance, its difficult to get the lag between the east coast of America to London below about 70-80 milliseconds. Doesn't sound like much, but it all adds up if your requestor is making hundreds of requests. CDNs get around this my placing the data people may request in many different locations around the globe, and then pointing requestors to the nearest copy. The biggest CDNs claim to run networks on the "edge of the internet", where through partnerships with consumer ISPs they serve content directly from the ISP's own network.

So, I had a few hours spare the other weekend and decided to take them up on their free trial month offer to ease the load on play.tm. Sign up was easy, and the account was setup instantly - FTP, admin tools and the hostname (which is ferrago.cachefly.net). Very impressive. So I got straight on to FTPing up the files.

I'd decided to put the simplest of the site's files on Cachefly - all of the CSS, Javascript includes and static page furniture images. This stuff doesn't take up much disk space (a large cost at Cachefly as they replicate the files onto multiple sets of storage) and doesn't change very often - yet it makes up a very large proportion of the volume of HTTP requests to site. It was a quick win to FTP it all up, then change some variables in the site's config files. The only real negative point on the hosting came up here - in order for the files you're FTPing up to become available at your hostname you have to end the FTP connection. This is annoying, and I'm not sure why you have to do it. It caught me out FTPing one set of files - and I couldn't figure out why they didn't become available.

The stats tools provided by Cachefly are good - showing all the data you would expect from log files, in a useful interface. I love the POP view that lets you see where your files were served from. Its all live information as well, which is impressive for such a distributed system.

In summary then - it all works rather well. The choice of the files to put up was simple given the high prices we would have been paying for them to host the hundreds of gigs of content images and videos. The stats show that a large proportion of hits are served away from our London server farm, so the benefits are obvious. I recommend their service without hesitation.

So, I bought a MacBook

The other month my dodgy old Dell laptop (purchased for £250 many years ago) was feeling the pain, with numerous physical and performance problems. It served me very well especially considering it was only had a 1.2Ghz chip and 256mb RAM, plus the harsh environments it was subjected to - ranging from rubbish Thai power supplies, an Asahi-fuelled tour of Japan, BarCamps, trains, planes and several datacentre floors. In it's favour was that it was very small and I didn't care much about it getting scratched or broken, on the downside was the shockingly poor battery life and the annoying sticky-outty wifi card that had to be installed.

It was time for it to go as it simply didn't cut it anymore.

My mate Luke and I trundled off to the Apple store to have a look at what was on offer and walked out with a couple of the £749 white Macbooks (minus 6% BBC discount).

Overall, I'm very impressed with it. Very impressed indeed. The overwhelming feeling is that everything just works - wifi, webcam, software installs the lot. I'm also surprised how much software I play with on Windows has a Mac version. Firefox and all the handy extensions work, Skype (version 2, but not 3 yet), Foldershare, Terminal Services client (with minor limitations), MySQL GUI Tools - the lot.

Battery life is excellent, the close-lid standby is quick and easy. WinXP's standby and hibernate never seemed to work correctly for me - leading to distrust and me never using them. The power cable design is in two pieces allowing you to leave part under the table rather than scrambling under there to unplug it - just little things like that make it so much easier to use. Also, while we're talking about the power cable - the magnetic connector has already saved a huge repair bill from me being clumsy in a Hong Kong hotel room.

Wifi and bluetooth just work - a revelation for a Windows devotee. Dead easy to use with zero hassle. Very impressive.

There doesn't seem to be much freeware around, but cheap independent software can be bought for less than $20. This is often produced by a one man show, and is of an excellent quality. As far as OS X goes - I'm surprised by the features I use. The F9 to F11 window moving keys are superb, but the dashboard is too slow and clumbersome to be of use to me. Saying this though - the MySQL monitor widget that came with the MySQL GUI tools is very cool - even though it refuses to remember my login details. FTP in finder works a treat, and the help files are useful for a Mac newbie such as myself.

So - I'll probably give another update in a few months time, with any teething problems. Jason on Mac out.

SVG thoughts

Is it me or is the development of SVG features in browsers painfully slow? Whilst Flash continues to seemingly take over the world delivering insane amounts of content for YouTube, MySpace etc, SVG is little more than a tech demo with people going nuts over a tag being put into Gecko 1.9 alpha releases.

Separation of Content, Design and Behaviour - when it goes bad

So, you know the deal - anything a client side developer (HTML monkey, Javascript hacker, CSS stylista... whatever you call yourself) produces should be divvied up into separate technology specific files - with the only references between those files being the includes, and the DOM references.

The benefits of this approach arn't immediately obvious but the learning curve is small (we all use these technologies most days if your in the biz - its just structuring your work differently) so the only change you need to make is a cultural one. Once your done though - you can start separating your workforce into more focused disciplines, allowing specialisation to improve the quality of your overall product. Not to mention that when a marketing bod/boss/designer asks for a change you should only have to fiddle with one bit of code - coding nirvana.

This technique (or techniques, depending on how you look at it) is covered very well in these A List Apart articles:

http://alistapart.com/articles/behavioralseparation
http://alistapart.com/articles/separationdilemma

So, how does this work in relation to a CMS?

In a standard CMS, all of your 'content' is held in a structure (be it XML, a database or some other kind of flat file) then it is transformed out when published onto whatever platforms you desire - WML for crap mobiles, HTML or XHTML for the browser, RSS for your feedreaders, CSVs for use in spreadsheets, XML for other syndication etc etc.

Problems occur when you then try and incorporate presentational data in that 'content'. In a CMS I've worked on one requirement was to have the content editor be able to position images and quotes around paragraphs of the body text of a page. This required the content to contain presentational information - the content editor's layout choices. So the data that ties quote 2 with paragraph 4 (for example) is regarded as content, but is actually just a presentational detail.

The XML looks something like this (very simplified)...

Not ideal.

It gets worse when you then have to publish this to a page. Like good little developers we put all our presentational information in the CSS - no problem. The CMS then pulls in the correct CSS for the correct page. No proble... uh ho - how does the CSS know where to put paragraph 3's quote? We could use an inline style or add a class to the quote, which the CSS would use to put it left or right.

The class solution is the obvious frontrunner - it increases cacheability as well as being more maintainable. Here the debate begins: what do we call the class? Given that it will only ever be used to move an item to the left, the obvious choice is calling it "left". However as we all know - a class or ID reference shouldn't imply the style that is placed on it. Taken to the extreme - if you call your class "red", then what happens when your marketing bod/boss/designer asks for it to be green. You're in trouble.

So candidates for these two class names put forward were...

"left" and "right"
"l" and "r"
"position1" and "position2"
"p1" and "p2"

Which is best? Here is the disappointment - I don't know. On one hand you have the maintainability of "left" and "right" - any developer approaching the code knows what is going on straight away. However with this method you're shooting yourself in the foot for alternative versions of the page or if the change comes in to move images on the right to the left or vice-versa. The argument for ignoring the "not implying style in a class/id" rule is that any left/right changes should happen in the XML or transforming language, not the CSS.

I lean towards calling it "left" and "right", but I can see both sides of the arguement. In the first sites we did using this XML, this is what we did, but then it got changed to "position-a" and "position-b" in a subsequent release of the XSLT.

BarCamp London 2006

Went to this the other day. Lots of fun. Checkout the coverage.

MetaWeather Beta Launch

My blog has cooled down a bit recently (as if it was 'hot' beforehand), as I've spent most of my spare time moving house and getting MetaWeather going.

The concept is pretty simple. It's a weather data aggregator, that goes out and acquires weather information from sources (primarily XML, or all XML, depending on how you look at it - more later) and then calculates the average weather.

Seemed nice and simple on the face of it, and it was to a degree (weather pun intended), however were are some complexities I encountered...

Datasources - getting XML

The first problem to solve was acquiring the data. This is done by consuming XML over HTTP from various outlets over the web.

One source I particularly wanted to use data from was the BBC, mainly because I work for them, and also because I'm interested in the backstage.bbc.co.uk initiative. Problem is that the BBC don't obviously publish their weather data as XML. I presume this is because of licensing issues with the Met Office - but they continue to assure backstagers that it is coming. Quite a few apps available around the web use BBC weather data, the biggest of which appears to be the Konfabulator widget (mentioned on the backstage.bbc.co.uk site here). A quick shifty through the code (which turns out to be some nice digestible javascript) reveals that they are in fact consuming the BBC's WAP site for data.

Now, I think WAP is crap (so does Jacob - the buying a newspaper bit is hilarious) and I've never used the BBC WAP site before (I have however used the far superior mobile site). It seems WAP is valid XML, and contains quite a bit of weather data. It needs some string manipulation to get the readings out, but most of the legwork can be done via XPaths.

Weather.com has well thought out web service, although the registration on their site is a little hard to find - probably because they aren't making any money from it (or at least, not much). Not much to say there - it just works.

Weather.gov is a site run by NOAA who appear to the equivalent of the Met Office, on steroids - presumably because they actually have weather (proper weather, we're talking city levelling stuff) in the US. As opposed to perpetual drizzle over London. NOAA offers a variety of XML goodies via their site, including an interesting interpretation of the use of RSS (if not exactly per spec, I'd argue) in this feed system. Weather Underground presents its data in a similar way.

I've not found any other free sources yet, but I'm working on it. Accuweather is another large site, but it appears they have a view on free weather data that isn't exactly consummate with the aims of this project.

Standardising the data

Once I've got hold of the data sources the problem of storage occurs. We'll have to standardise the data so we can store it, then later compare it.

The data is recorded with a 'fetch date' and an 'applicable date'. Data with the same or similar (within a specified time period) 'fetch date' and 'applicable date' are called 'observations'. Readings with an applicable date in the future are therefore 'forecasts'. I suppose data with an applicable date in the past could is archived data. As its not much data we're talking about (and I have a fair amount of database juice to play with) I'm not destroying any data.

Most of the measurements are pretty straightforward to standardise. Temperature is held in centigrade, wind speed in MPH, wind direction in degrees - all fairly easily converted from various formats in which the sources are delivered - Fahrenheit, KPH or a compass point as examples. I'll probably post all these conversion functions online at some point.

However the most important data is the hardest to tie down - what we've called the 'Weather State'. This is often presented to us as a short string such as 'Showers' or 'Sunny', but more often than not its more complicated. Most weather sources offer an icon to go with this data - a much more machine readable and workable way of dealing with this problem than text designed to be read by a human. It is this data that we use to determine the weather state.

If I was an IA (information architect), I'd be referring to the different sets of source's data as 'controlled vocabularies'. MetaWeather has its own controlled vocab, and we also hold a big fat thesaurus. This thesaurus is a mapping of the different source's entire vocab to MetaWeather's. So for example... WeatherUnderground may call a particular weather state "Light Snow Showers" however MetaWeather will simplify this to just "Snow" in our controlled vocab. MetaWeather does sometimes take into account states that may be in between two in it's controlled vocab. For instance - a simple example - "Showers" is between "Clear" and "Rain". However, I'm not an IA, and this is a really simple example of controlled vocab with not a lot of data.

Some problems arise with this when you get to the nitty gritty of some source's data. For instance a source may return "Heavy Volcanic Ash", "Unknown Precipitation" or something equally crazy. In this case, we just disregard the weather state from this source - after all we have other sources to get data from, and these states don't occur very often.

Average Wind Direction

Wind direction is held in the database in degrees, and as such represents a problem in calculating the average. For example if one forecaster predicts a 10^o wind and another a 350^o gust then the mathematical average will be 180^o, not 0^o.

The solution was to combine the wind speed with the wind direction and convert it into a vector. Painful attempts to remember GCSE maths lessons ensued along with roping in of workmates to explain the vagaries of radians and trig functions. I've still to understand this fully in the context of this project, and therefore haven't implemented it yet. The rubbish average wind direction works for the moment, with a note explaining its inaccuracy.

Predictability

Predictability is a measure of how different forecasters predictions of the weather are. The calculation is pretty straightforward, and just involves getting the standard deviation of the weather state values and converting that into a percentage. Eventually I'll hope to get a whole weather predictability as opposed to just the weather state, by averaging the percentage deviations across other readings, which will probably give us a better view on the predictions.

So thats it for now. I've got a lot more to do on this - loading more locations, more sources and probably knocking up an API to present all the data.

Common RSS Mistakes

The ubiquitous syndication format RSS is based on the XML file format, and as such has some strict rules that need to be followed to maximise its usability, to stick to guidelines, and even just make it work. Here are some common mistakes...

MIME type
MIME types are rarely an issue that a web developer comes up against, as it's often handled without user intervention by the web server software. Your standard Apache or IIS config will include mapping for common file formats to their MIME types. In Apache this will usually be done through the mime.types file and in IIS it's covered in the IIS Manager properties, or the properties of an individual website.

However, RSS is often served...

Through a scripting language like PHP or ASP
With an unrecognised and not configured by default format like ".rss"
Without a file extension

In these cases the page is often served with a MIME type other than "text/xml" as it should be done. The result of this is that the page is displayed as plain text. Not a big deal, but not ideal.

The solution depends on how your generating your RSS feeds. If your generating them on the fly from a script then the fix is easy. In PHP use...

header("Content-Type: text/xml");

In ASP or ASP.net use...

Response.ContentType = "text/xml"

If your generating the feed to the server then you can just save the file with the file format .xml, then this (in Apache and IIS at least) will be served with "text/xml". If you can't save the file with this extension, then your going to have to find out how to set the MIME type correctly for the extension you choose.

Content in description
This one really gets on my nerves. The RSS 0.91, RSS 1.0 and RSS 2.0 specs clearly state that the description node should be filled with a "synopsis", or "brief description/abstract" of the item, rather than the whole content of the item. If you really want to stick the whole content in your feed then use as described here. Arguably you shouldn't really put HTML in the description either. However nobody says you can't, I suppose.

Lack of links to the feed
You should advertise your RSS feeds in a couple of ways...

Using the de facto standard bright orange button
Putting the construction in every page on your site, or at least just your homepage

Not using and
Not really a mistake, but people should definitely make use of these elements more. I see people putting comment counts, and category names in the . Very annoying.</p> </article> <article> <h1>The Tim Westwood Drinking Game</h1> <p><a href="http://en.wikipedia.org/wiki/Tim_Westwood">Tim Westwood</a> presents the rap show on <a href="http://www.bbc.co.uk/radio1">Radio 1</a> on Friday and Saturday nights. Alledgedly the inspiration behind Ali G, he is a white guy immersed in the predominately black culture of hip-hop and rap music.</p> <p>Tim's catchphrases, and his show's timeslot make it perfect for a drinking game.</p> <p>Rules of the game are simple. Take a drink (shot or swig) whenever...</p> <ul> <li>Tim says any of the following... <ul> <li>"Man this joint is crazy"</li> <li>"Lets go baby"</li> <li>"Damn, thats what I'm talking about"</li> <li>"Damn, thats the way it goes down"</li> <li>"Its a good look"</li> <li>"Its a bad look"</li> <li>"That's what we does"</li> <li>"Understand that"</li> </ul> </li> <li>Tim pimps a gig he is doing in a seemingly random, un-hiphop, crap town such as Swindon, Banbury, Slough, or Enfield</li> <li>Tim plugs a TV show he is on</li> </ul> </article> <article> <h1>DIVless rethinking</h1> <p class="info">This is a follow up to the post made previously: <a href="/blog/entry/2005/10/why_a_divless_page">Why a divless page</a></p> <p>When I first developed the HTML for this blog, I used no DIVs. Several things have happened to make me reassess this approach since I wrote the HTML and blog entry.</p> <p>The first was in the development of the HTML, when I realised that I couldn’t put H tags in the DT. This was a big problem as most pages (particularly the homepage) contain a lot of text, and breaking this up into navigable areas for screenreaders should have been my aim. In the end I ignored the problem, and left it as it was. I did have a fairly decent H structure anyhow, although not for the bulk of the interesting bits of the page.</p> <p>The second thing to happen was a conversation with a work colleague. He suggested that a list of blog entries were not a list of definitions. Now, I’m still unsure about this. The W3C says that a DL...</p> <blockquote><p>consist of two parts: a term and a description</p></blockquote> <p>A blog post title and the post's text? A recent visit to my workplace by <a href="http://www.molly.com">Molly(.com)</a> suggested otherwise, however. <a href="http://www.stuffandnonsense.co.uk">Andy Clarke</a> described how he had successfully used a definition list in a product listing. Using the DT as the product heading, then the product’s image and details in DDs. So, I guess this depends on your interpretation of the standard.</p> <p>I also had an email from a guy called Frank Olieu. He took me up on the statement in which I suggested that you should use some DIVs for the basic layout of a page. Here is what he said...</p> <blockquote><p>I just bumped into your divless page, which I found refreshing and inspiring…</p> <p>But allow me a question...<br/> You wrote:</p> <p>(...) The solution is to use divs only when semantically meaningful markup is not possible. For example, if a design calls for a fixed width centred design, your going to have to wrap the lot in a div. (...)</p> <p>... If I get it right, you could achieve the same thing by styling the /body/ element itself, even with borders if required. The only drawback I can think of, is that it would probably not work in anything prior to IE 5.5... I've been making some experiments with divless layout for some time, using a styled body as the main container, and the browsers (*) I tested with could cope with that.</p> <p>Here is a "quick and dirty" mock up : <a href="http://www.design.olieu.net/files/divless.html"> http://www.design.olieu.net/files/divless.html</a>, where even the /html/ element is styled! (it's partly in Danish but easy to figure out...)</p> <p>What is your opinion? Would it be "safe" to use in real life?</p> <p>(*) browsers: Firefox 1.07, IE6, Opera 8.5, Konqueror.</p></blockquote> <p>Well, some browser support not withstanding, it seems that you can style up the HTML and BODY tags enough to produce <a href="http://www.design.olieu.net/files/divless.html">a decently laid out page</a> without the DIVs I guessed you would need. Good work Frank!</p> <p>Back to his question though - is it 'safe' on a public website? My answer to this was "it depends". If it's your techie focused blog your doing it on, then by all means go right ahead and do it, after testing the hell out of it. It's a nice little tech demonstration. However, I did say that he probably shouldn't use it for any much more than this. If your developing the next Google, a consumer e-commerce site (reliant on sales conversions) or any other kinds of mainstream web resource, then I would steer well clear. Best stick in a few DIVs - the cost of losing 5-10% (at a guess, perhaps more?) of your visitors because the page renders incorrectly (or worst case, not at all!) is too high when the only benefit is a bit less code and the kudos of pulling it off!</p> <p>So, what was the verdict on this matter? Did I relent and stick some DIVs in? Kind of. </p> <p>My homepage was getting too big (over 30k of HTML, and growing). I had not really thought out the navigation, and all the blog entries full text was clogging up the page increasing download time and obscuring the content right at the bottom. Because most (all?) of my posts are not timely, as in it doesn’t really matter which order you read then in, I figured this was a bad thing. I also had the screenreader issue mentioned above.</p> <p>The solution therefore was to leave the 'Blog' section of the page as a definition list and only place the first paragraph on indexes. This, I believe makes the blog list much more like I think a definition list should be. This approach also got around an IE bug where the text was going over the left boundary of the DD if there were lots of long DD’s in the page. Very strange, but no longer happening.</p> <p>So, no DIVs yet - but I'm certainly thinking about it for the blog entry pages.</p> </article> <article> <h1>Absolute positioning ain't that bad</h1> <p>Absolute positioning has a bad rep from its misuse in the early days of CSS, and misunderstanding of how it works</p> <p>The most common CSS misinterpretation I’ve seen is the fallacy that everything absolutely positioned must be positioned relative to the body of the page - 0,0. The fact of the matter is that an element is absolutely positioned relative to the nearest <code>position:relative</code> parent. I guess people came to this misunderstanding through not having any relatively positioned elements on the page, and then attempting to use absolute positioning. For the record, I’ve made this mistake. If you look hard enough you’ll find it still on the net - although the page does work well, just with huge pixel values!</p> <p>So therefore cleverly laid out elements positioned relatively (I'm thinking the general page layout), and provide a structure for the absolutely positioned elements below it to hang off.</p> <p>One other misunderstanding is that you have to use pixels to absolutely position an element. Not so - em's work well, as do percentages.</p> <p>One problem I've encountered with absolutely positioned elements is the uncontrollable length. I've often found it difficult to position something underneath an absolutely positioned element because of the seemingly uncontrollable tendency of the content inside to overrun and overlap the content in the element below.</p> <p>An upside of absolute positioning I’ve thought of is the rendering speed. Having absolute values must surely give more hints to the client as to the position of elements. I'd imagine this isn’t completely true if these elements are inside a relative element, where that element’s position isn't known until the page is displayed. Someone is bound to correct me, but it must help right? Anyhow - rendering speed isn’t that much of a consideration with modern browsers and computers when taken into consideration with the latency of the delivery of the page.</p> </article> <article> <h1>Gaming: Persistent scoring, and fair team play</h1> <p class="info">Article first appeared <a href="http://www.ferrago.com/story/6839">here</a>, and was edited by <a href="http://www.ferrago.com/author/Luke%20Guttridge">Luke</a> </p> <p>Very rarely do I get addicted to a game, but Battlefield 2 has me well and truly hooked. Team Fortress Classic, Counter-Strike (and CS: Source), Simcity, Half-Life 2 have all had me in their clutches, but this addiction is worse, much worse - its based around an MMORPG levelling and rewards system. Now I know what the guys who call Everquest 'Evercrack' are trying to weans themselves off.</p> <p>Battlefield 2 requires you to register with EA, and in return they track your progress in ranked online games awarding you ranks for general achievement, and medals for training on specific tasks. All this behind the scenes technology is run by Gamespy, who interestingly offer up all of your stats in an XML format. A variety of tools (such as <a href="http://www.bf2s.com">BF2S.com</a>) are available online to consume these stats outside the game. Very cool indeed... the techie in me loves this. Here is my <a href="http://bf2s.com/player/jasoncart/"> woeful combat effort</a>. This competitive, rolling, MMORPG-like centralised ranking and scoring system has its drawbacks though, especially in a game that requires teamplay to master individual maps.</p> <p>The most prized asset for the competitive, ranking-sensitive Battlefield 2 player are the helicopters. These fearsome death machines are almost ludicrously overpowered, in the right hands. They feature a gunnery position with a nasty cannon (best for troop takedown), and TV-guided missiles (best for vehicles). The pilot, whose job is to grapple with the newbie unfriendly control system, and powerful engine also has access to a bank of effective-in-quantity missiles. Newbies inevitably do get hold of them sometimes, to disastrous effect - they are pretty difficult to fly until you get your head around the control system. I'm reminded of a driving school car around my way that says "Everyone had to learn once!" on the back.</p> <p>The sought-after nature of these monsters causes the spawn spots beside them to be crowded with hopeful 'copter pilots who proceed to the heli spawn point and tap E (the enter vehicle key) rapidly. Some take it further. Teamkilling is prevalent in Battlefield 2, and most of the deliberate attempts happen around the helicopters. The most common form appears to be a player in the sniper class killing the person inside the chopper, than taking their place. Other methods of removing a player from the valued war machine is to take the heli to the edge of the map and dip the 'offender' out of the combat area. This causes them to lose health and eventually die, enforcing the map rules. Very clever, but very annoying. Other techniques range from 'verbal' abuse over the text or voice chat, and initiating pointless (more on that in the next imminently) kick votes.</p> <p>What makes this situation worse is the frankly awful in-game kicking system. Most decent Counter-Strike servers have an admin around to call upon, or actually witness in-game offences, and failing that a decent service to report offenders instantly or after the game. Not so in most BF2 servers, particularly the official ones, it must be noted. The game relies on a stupidly flawed voting system to give griefers the boot. I've never seen this work, ever, even on the most blatant teamkillers.</p> <p>This system requires a certain percentage of the server's population to answer a poorly displayed "ALL IN CAPITAL LETTERS" question. This percentage (I'm unsure as to the actual amount) usually means that twenty-odd people (on a 64 player server) must understand and respond to the kick vote in a certain amount of time. It never happens - I've never seen someone kicked. Of course, getting away with the 'crime' makes the cocky little blighters more confident, and the behaviour spreads like wildfire as people see the opportunity or defend themselves.</p> <p>I mentioned previously that the game requires teamwork to conquer the map. This is only true of course if the opposing team are not in the same disarray as yours! Squads of different classed players is undoubtedly the way EA designed the game to be played. 'Support' and 'Medic' class troops are designed to backup the heavy hitters, and they do so well - getting rewarded with points for healing and re-supply work in the same way as the guys with the big guns get them for doing the messy work on the front line. A good working team will, however, probably never achieve the kinds of scores a skilful heli pilot and gunner will, hence the competition for the choppers. This is a real disappointment for the people who try to play the game as it is meant to be played (no Nvidia bungs here, honest).</p> <p>The other trouble is people 'harvesting' ranking as you may expect people to do in MMORPGs. These exploits mainly revolve around the 'Support' and 'Medic' classes, where two people on a server deliberately kill each other, then use the medic tools to revive them and score points - repeatedly. These issues were thankfully addressed in the latest patch (1.03). This kind of behaviour is undoubtedly linked to the scoring system. Having your point score stored for eternity and being ranked in full view of the rest of the game (or indeed the world, as you can see above) causes people to take the game a lot more seriously. This is great for EA, or the fair play minded gamers that make up 95% of the server users - I know I enjoy being 'promoted' to the next rank. However this TKing minority, coupled with poor controls on player kicking, spoils the game for everyone. Gaming (some would say hacking, or 'working') the game and its rules is part of the fun, to a degree, but when I sit down for a nice evening-long game of BF2, I don't expect to have it wrecked by someone else, just like I don't expect it when sitting down to watch TV or consume other forms of entertainment.</p> <p>Where next for MMORPG-like scoring in other genres, then? I'm getting nervous already...</p> </article> <article> <h1>What Flickr Needs</h1> <p>Flickr is without doubt the best 'web application' I've used. The point on which a web <em>site</em> becomes a web <em>application</em> is a little unclear to me at the moment, but we'll stick with application for the time being. It seems the gratuitous use of AJAX tech is the main driver for HTML based tools to be called web applications these days - Web 2.0 people tell me :-)</p> <p>So what is so great about it huh? Well the main (and somewhat fluffy) overall reason its so good is the style is purveys. You know these guys are a class act - every part of the design and interface is almost (more on that later!) perfectly crafted. Its also rather fast, particularly on a broadband connection. I dread to think of the infrastructure these guys have in place to handle the crazy numbers of images and transactions they are dealing with. All this is for a the measly fee of $25 (£14 odd) a YEAR for the crazy 2Gb upload a month.</p> <p>However all is not perfect in the land of Flickr. Regular 'Flickr Massages' occur, where the site is taken offline, presumably for database maintenance. Since being bought by the behemoth Yahoo Inc things have got better but they still happen too often. Even with some of the <a href="http://jeremy.zawodny.com/">finest practical MySQL minds</a> around they can't keep the app going 24/7. There are also features missing that users have been clamouring for....</p> <p>Sets of sets is my pet peeve. Most people have large sets (custom groups of photos) of, in my example, photos of a particular holiday. These become unwealdingly large and pretty difficult to navigate after you have more than say 50-100 photos in them. The solution therefore is to break them up into a nested hierarchy of sets with, for example, "<a href="http://www.flickr.com/photos/jasoncartwright/sets/750984/">Thailand 2005</a>" having subsets of "Bangkok" and "Phuket" etc. Purists argue that you can use the tagging system can be used to do this - a search for <a href="http://www.flickr.com/photos/jasoncartwright/search/text:bangkok+thailand/">"Thailand Bangkok"</a> solves the situation, they say. Whilst I do like the tagging system - it doesn't cut it for this task of arranging my photos, by my definition, to the potential audience. Yes people can search - but I want to build a structure that shows them through the photos as well. Stewart (Flickr co-founder) has replied to a <a href="http://www.flickr.com/forums/ideas/2078/">monster thread</a> on the Flickr forums stating that the feature is coming.</p> <p>My other want of Flickr is better panoramic image handling. It seems that many cameras come with software packages that allow the stitching of multiple images to form panoramas. Some cameraphones (including all new SonyEricssons from the K700i upwards) come with panorama creation on the fly, with some very <a href="http://www.flickr.com/photos/evilcoffee/search/tags:k750i/">impressive results</a>. My stitching software of choice is <a href="http://www.cs.ubc.ca/~mbrown/autostitch/autostitch.html">Autostitch</a>, which works a treat, as <a href="http://www.flickr.com/photos/jasoncartwright/sets/925625/">you can see</a>.</p> <p>So, what can Flickr do to help? When you upload to Flickr it creates several different versions of the image in different sizes. Now, these work great when the image is a standard 4:3 photo (which 95% of photos on Flickr are, I'd imagine). However things get a bit out of shape when you stick images on there that are much, much wider than they are higher. By out of shape, I don't mean distorted (they are far too clever for that), although some images appear to be of a slightly <a href="http://static.flickr.com/30/42249337_6e05e07397_s.jpg">overly optimised</a>. The problem occurs when you get to the photo's page - the image is <a href="http://www.flickr.com/photos/jasoncartwright/42249337/in/set-925625/">pretty unusable</a> due to the weird dimensions.</p> <p>Flickr used to use a flash interface on this photo page - and I propose they should bring it back, or make it optional or automatic per account, image set, or individual image. The flash app could load a slightly larger version of the image, and allow the viewer to scroll around inside it. If they think they have moved away from flash for displaying images like this, then a CSS based solution could be possible - with a horizontal scrollbar (bit of a usability no-no I know) and/or some clever image moving javascript. </p> </article> </main></body></html>