Five months ago I wrote a Twitter bot called @whensmybus. It took me a fortnight to code up and test the first version, which was pretty simple to begin with – it took a single bus number, and a geotag from a Tweet, and worked out when the next bus would be for you. And then people started using it, and really liking it. And because they liked it, they found ways of making it better (curse those users!). So I had to add in features like understanding English placenames, being able to specify a destination, handling direct messages, multiple bus routes, and tolerating the many many ways it’s been possible to break or confuse it, and this took up a lot of my time. And it was fun, to be honest.

At the same time, those bloody users also asked me when was I going to do a version for the Tube. But I was too busy adding the features for @whensmybus, and that’s one reason why it took me five months to write its counterpart, @whensmytube, which I launched last week. But there’s a stack of other reasons why it took so long. It didn’t seem too difficult to begin with. Just like with buses, Transport for London have made their Tube departure data open-source (via a system called TrackerNet), as well as the locations of all their stations. It would be pretty simple to do the same for tube data as it would for bus data, right?

Wrong.

So, for anyone interested in open data, software development, or just with a lay interest in why software doesn’t get new features quickly, here’s a run-down of why:

1. The Tube data isn’t complete

TfL helpfully provide details of all their Tube stations in a format called KML, from which it’s reasonably easy to extract the names and locations of every station. Well, they say “all”. That’s a bit of a lie. The file hasn’t been updated in a while; according to it, the East London Line is still part of the Tube network, and Heathrow Terminal 5 and Wood Lane stations don’t exist; neither do the stations on the new Woolwich Arsenal and Stratford International branches of the DLR. This has been griped about by other developers, but no update has been forthcoming. So it took time to do the ballache task of manually writing the data that hadn’t been included in the first place.

To make things more annoying, certain stations are left out of the TrackerNet system. If you want live updates from Chesham, Preston Road, or anywhere between Latimer Road and Goldhawk Road on the Hammersmith & City, you’re plain out of luck. Sorry, this is TfL’s fault and not mine. This also wasn’t documented anywhere, just omitted from the system documentation.

2. The Tube data isn’t built for passengers

To be fair to TfL, they do say what the TrackerNet service was meant for – it is built on their internal systems and was for non-critical monitoring of service by their staff, and there is a disclaimer saying this. The public version is useful, but unlike its bus counterpart there is a lot of data there which is not for public consumption. If anything, it’s too useful, as it contains irrelevant information such as:

• Trains that are out of service or “Specials”
• Trains that are parked in sidings
• Trains on National Rail systems, like Chiltern Railways, that run over Tube lines
• Data on whether a train is scheduled to go to a depot after its journey
• Some trains just don’t know what their final destination is yet, and are just labelled “Unknown”

And none of these special cases are documented in the system. So I had to spend a lot of time working out these odd edge cases and filtering out the chaff. And the code is by no means complete – I have to wait until irrelevant information is shown up to be able to filter it, because TfL don’t provide anywhere a list of possible values. This is annoying – so much so that I have even taken the step of submitting a Freedom of Information request to find out all the possible destinations a train can be given on the system to make sure, but I’m still waiting on it.

The documentation also falls down on being useful to reuse. For example, each station has a name (e.g. “King’s Cross St. Pancras”) and a code (e.g. “KXX”). Because spellings can vary, it’s easier to use the three-letter code when doing lookups for consistency. But the list of codes, and the station names they correspond to, were locked in a bunch of tables in a write-protected PDF, so it was impossible for me to create a table of code-to-station-name lookup table. In the end, I’m glad that someone had done the hard work for me, rather than I having to manually type them out.

On top of that, the system uses terminology more suited to insiders. For example, most stations have platforms labelled Eastbound/Westbound or Northbound/Southbound, which is fine. But the Circle Line and the Central Line’s Hainault Loop have designations “Inner Rail” and “Outer Rail”. And then to make my life even worse, some edge cases like White City and Edgware Road have platforms that take trains in both directions. This is confusing as hell, and so I had to spend a bit of time dealing with these cases and converting them to more familiar terms, or degrading gracefully.

This is a pain, but worth it. As far as I’m aware, no other Tube live data app (including TfL’s website, or the otherwise excellent iPhone app by Malcolm Barclay, which I regard as the gold standard of useful transport apps) takes this amount of care in cleaning up the output presented to the user.

3. Humans are marvellous, ambiguous, inconsistent creatures

And then on top of that there’s the usual complications of ambiguity. There are 40,000 bus stops in London, and typically you search for one by the surrounding area or the road it’s on, because you don’t know its exact name, and the app can look up roughly where you are, and give an approximate answer. But, there are fewer than 300 Tube stations, and so you’re more likely to know the name of the exact one you want. But, there are variations in spelling and usage. Typically, a user is more likely to ask for “Kings Cross” than the full name “King’s Cross St. Pancras” – punctuation and all. This all needs dealing with gracefully and without fuss.

4. Despite all my work, it’s still in beta

There’s plenty @whensmytube doesn’t yet do. It only accepts requests “from” a station and doesn’t yet accept filtering by destination “to”. This is because, unlike bus routes, most tube lines are not linear (and some even have loops). Calculating this is tricky, and TfL don’t provide an open-source network graph of the Tube network (i.e. telling us which station connects to which), and I haven’t yet had the time to manually write one.

5. But I’m still glad I did it

Despite all my problems with wrangling TfL’s data, I’m still pleased with the resulting app. Not least because, hey, it shipped, and that’s to be proud of in its own right. But more because everything I learned from it has kept me keen, and it’s had some pleasant side effects. The refactoring of the code I had to do has made @whensmybus a better product, and all the learnings of how to deal with the Tube network meant I was able to code and release a sister product, @whensmyDLR, with only a few days’ extra coding. Not bad.

But, here’s some quick conclusions from wrangling with this beast for the past five months:

1. Open data is not the same as useful data If it’s badly-annotated, or incomplete, then an open data project is not as useful. Releasing an API to the public is a great thing, but please don’t just leave it at that; make sure the data is as clean as possible, and updates are made to it when needed.
2. Open documentation is as important as open data It’s great having that data, but unless there’s documentation in an open format on how that data should be interpreted & parsed, it’s a struggle. All the features should be documented and all possible data values provided.
3. Make your code as modular as possible If you’re having to deal with dirty or incomplete datasets, or undocumented features, break your code up into as a modular a form you can get away with. The string cleaning function or convenience data structure you wrote once will almost certainly need to be used again for something else down the line, and in any case they shouldn’t clutter your core code.
4. In the end, it’s worth it Or, ignore all my moaning. Yes, it can be a pain, and annoying to deal with cleaning up or even writing your own data to go along with it; but in the end, a cleanly-coded, working product you can look on with pride is its own reward.
5. Thank you TfL Despite all my bitching above, I’m still really grateful that TfL have opened their datasets, even if there are flaws in how it’s distributed and documented. Better something than nothing at all – so thank you guys, and please keep doing more. Thank you.

But people don’t just take buses in London. They also take the Tube. And so it only seems fair to build a sister service for @whensmybus for the subterranean-inclined. So, introducing… @whensmytube. It does the exact same thing – taking advantage of Twitter’s realtime and geolocation capabilities and mashing them up with TfL’s open APIs to give you live Tube departure times for nearly any station on the Underground. Just Tweet:

@whensmytube Central Line

with a GPS-enabled Tweet, or:

@whensmytube Central Line from Liverpool Street

with an ordinary Tweet. More information and a full description of its abilities and how to use it are available here. Please use it! And break it! It’s still in beta, and any feedback would be much appreciated, thank you.

But, hang on. That’s not all! There’s not just the Tube in London. There’s also my beloved Docklands Light Railway. And it would be cruel to leave it out. So have two for the price of one – if you’re a DLR lover, please try @whensmyDLR for size as well:

@whensmytube DLR

with a GPS-enabled Tweet, or:

@whensmytube DLR from Poplar

with an ordinary Tweet. Like its Tube and bus counterparts, it’s reasonably flexible, so please check out out the help page. And please give any feedback you can, thank you!

In higher education, although universities such as Bournemouth are praised by employers for working closely with industry, other universities and colleges have been criticised by businesses for running a significant number of “dead-end” courses in computer science, with poor prospects of employment for those enrolled.

And from my own anecdotal experience, that’s correct. For one reason or another, I’ve been reviewing CVs and interviewing people at work for developer roles last couple of months, and some of them were awful. They tended to have degrees or other qualifications from mid- and lower-tier universities and colleges, but had trouble telling the difference between PHP and JavaScript code, or were unable to provide even stock answers to well-versed problems such as sorting.

(Feel free to call me out as a snob on this one; I read my Bachelor’s in Computer Science at Cambridge, one of the few universities in this country where the majority of the course is spent not coding)

Anecdotal though my own experience, and many of the quotes in the article are, the Guardian’s campaign is laudable and I back teaching kids code in schools. But there are two issues I have with the campaign – it’s not just teaching, and not just code that needs to be taught (or learned).

Firstly, “digital literacy” is as broad a term as “literacy” or “numeracy”, and there are a range of different issues at stake. Take this complaint in the above article:

Ian Wright, the chief engineer for vehicle dynamics with the Mercedes AMG Petronas Formula One team, said: “There’s definitely a shortage of the right people. What we’ve found is that somebody spot on in terms of the maths can’t do the software; if they’re spot on in terms of the software, they can’t do the maths.

versus:

Kim Blake, the events and education co-ordinator for Blitz Games Studios, said: “We do really struggle to recruit in some areas; the problem is often not the number of people applying, which can be quite high, but the quality of their work. We accept that it might take a while to find a really good Android programmer or motion graphics artist, as these are specialist roles which have emerged relatively recently – but this year it took us several months to recruit a front-end web developer. Surely those sorts of skills have been around for nearly a decade now?

versus:

In a highly critical report last month, school inspectors warned that too many information and communication technology (ICT) teachers had limited knowledge of key skills such as computer programming. In half of all secondary schools, the level many school leavers reach in ICT is so low they would not be able to go on to advanced study, Ofsted said.

Computer Science is not Programming, and Programming is not Web Development, and Web Development is not ICT. What we have is a whole spectrum of different demands and of different roles, all of which have technology in common but often little else; producing computer models for a Formula One team or CGI Studio is going to demand a PhD-level or near grasp of maths or physics, combined with knowledge of highly specialised programming. Developing a front-end for a website still demands a reasonable degree of intelligence, but also a wider knowledge of languages and coding, and a better appreciation of more subjective issues such as usability, browser standards (or the lack of them) and aesthetics. Meanwhile, being adept with ICT doesn’t mean you have to be a genius or be an expert in code, but it needs to be more than how to make a PowerPoint presentation; how to use a computer properly and not just by rote, how to be confident in manipulating and understanding data, how to automate tedious tasks, how to creatively solve a problem.

Today technology is integrated to our lives to a quite frankly frightening degree. Should that mean everyone has to learn how to code? No. Should it mean everyone have an understanding of the basics, an appreciation of what computers can and can’t do, and the ability to use that knowledge to solve problems by themselves? Yes. But making everyone code is not the answer, and to me the Guardian is taking a bit of a “if it looks like a nail” approach to the problem of digital illiteracy.

That said, from my experience of the graduate CVs I read, the teaching of coding, as a practice, does need to improve. University courses should be better assessed and monitored and the “sausage factories” closed. Teaching how to code should be integrated into related subjects such as maths and physics wherever possible (and it’s worth noting many places do this well already). It shouldn’t just be coding that is taught, but how to define a problem, to break it down, and solve it. If anything, that’s more important – programming languages and technologies change all the time (e.g. how many Flash developers do you think will be about in five years’ time?) but the problems usually remain the same.

Secondly, there’s a spectrum of challenges, but there’s also a spectrum of solutions. It’s not just schools and universities that need to bear the burden. As I said, coding is a practice. There’s only so much that can be taught; an incredible amount of my knowledge comes from experience. Practical projects and exercises in school or university are essential, but from my experience, none of that can beat having to do it for real. Whether it’s for a living, or in your spare time (coding your own site, or taking part in an Open Source project), the moment your code is being used in the real world and real people are bitching about it or praising it, you get a better appreciation of what the task involves.

So it’s not just universities and schools that need to improve their schooling if we want to produce better coders. Employers should take a more open-minded approach to training staff to code – those that are keen and capable – even if it’s not part of their core competence. Technology providers should make it easier to code on their computers and operating systems out-of-the-box. Geeks need to be more open-minded and accommodating to interested beginners, and to build more approachable tools like Codecademy. Culturally, we need to be treat coding less like some dark art or the preserve of a select few.

On that last point, the Guardian is to be applauded for barrier-breaking, for making the topic a little less mysterious and for engaging with it in a way I’ve seen precious little of from any other media outlet. And the page on how to teach code is a great start – it should really be called how to learn code, because it’s a collection of really useful resources. For what it’s worth, I wrote a blog post nearly three years ago on things on things to get started on – though if I wrote it today I would probably drop the tip on regular expressions (what was I thinking?).

If I had one last thing to add, is that all of the Guardian’s campaign, and the support from Government, is framed around coding for work. Which is important – we are in the economic doldrums and the UK cannot afford to fall behind other nations. But, at the same time, the first code a beginner writes is going to be crap, and not very useful. Even when they get to a moderately competent level, it won’t be very useful beyond the unique task it was built for. Making really good code that is reusable and resilient is bloody hard work, and it would be off-putting to make the beginner judge themselves against that standard.

We need to talk a lot more about why we code as well as how we code. I don’t code for coding’s sake, or just because I can make a living out of it. I code because it’s fun solving problems, it’s fun making broken things work, it’s fun creating new things. Take the fun out of it, making it merely a “transferrable skill” for economic advantage, will suck the joy out of it just like management-speak sucks the joy out of writing. It doesn’t have to be like that. Emphasise on the fun, emphasise the joy of making the infernal machine do something you didn’t think it was possible to do, encourage the “Isn’t that cool?” or “Doesn’t that make life easier?”. Get the fun bit right first, and the useful bit will follow right after.

Bugfixes are all very well, but… by and far away the most requested feature was to allow people to get bus times without needing a GPS fix, to allow use on Twitter via the web, desktop app or not-so-smartphone. And although using GPS is easier, and cool and proof-of-concepty, it’s plain to see that making access to the app as wide as possible is what makes it really useful. So, from now on you can check the time of a London bus by specifying the location name in the Tweet, such as:

@whensmybus 55 from Clerkenwell

This will try and find the nearest bus stop in Clerkenwell for your bus – in this case, the stops on Clerkenwell Road, which are probably what you’d want). The more precise the location given, the better; place names are OK, street names are better. It works great on postcodes and TfL’s SMS bus stop codes as well.

The geocoding that makes this possible is thanks to the Yahoo! PlaceFinder API, so my thanks goes to them for making a service free for low-volume use. (Aside: you may ask why not use Google Maps? Because Google Maps’s API terms only allow it to be used to generate a map, not for other geo applications like this).

So, play away, and let me know what you think. Of course, it may not always work – geocoding is tricky and not foolproof; if it doesn’t, please let me know in the comments here, or just ping me at @qwghlm on Twitter.

More information and FAQs can be found on the about page, and the technically-minded of you might want to check out the code on github.

But, it’s a bit fiddly. The main website doesn’t automatically redirect you to the mobile version if you are on a phone. If you type in a location, (e.g. my local Tube station, “Limehouse Station”), you have to pick a match for the location first (from two identically-named options), then a second screen asking you to find a bus stop, and then you get the relevant times. On a phone, it’s just feels fiddly and frustrating especially when I know my phone has GPS in it and knows my location anyway.

Update/correction There is, as it turns out, the ability to find by geolocation on the mobile site, it’s just on a mobile browser I just get the main website and don’t get redirected to the special mobile site, which means I never knew about it (thanks to Ade in the comments for pointing this out).

If only there was a mobile-friendly, geolocation aware, real-time way of fetching information. Oh wait. There is. It’s called Twitter. Twitter has geolocation allowed on Tweets (if you opt in) and an API to fetch and send messages, so we have a system set up already in place for our needs.

I owe a big debt of gratitude to Adrian Short, who wrote a Ruby script to pull bus times from TfL. TfL have not officially released an API for Countdown just yet, but Adrian found it, and it’s there and accessible – providing the data in JSON format for each stop. That use got me thinking – if that data is available and can be parsed quickly and easily, why not make a Twitter bot for it?

With that, @whensmybus was born, and is now in beta. Try it out now if you like. Make sure your Tweet has geolocation turned on (for which you’ll need a GPS-capable smartphone), and send a message like:

@whensmybus 135

Or whatever bus you are looking for. Within 60 seconds, you’ll get a Tweet back with the times of the next buses for that route, in each direction, from the stops closest to your location.

Why each direction? Specifying a direction is fiddly and ambiguous; bus routes wind and twist, and some of them are even circular, so “northbound” and “southbound” are not easy things to parse. The name of your destination can have ambiguous spellings, and I haven’t yet got round to tying it in with a geocoding service like Google Maps. So, at the moment the bot simply tells you buses in both directions from the stops nearest to you. I might change this in future, once I’ve got my head around geolocation services and fuzzy string matching and all that.

It’s still beta (thanks to an early unveiling by Sian ;) ) and I plan in future to add enhancements such as the ability to use without GPS. I also need to write some proper documentation for it, and stick the source code on Github later tonight once I am home. The source code is now available on github, but do bear in mind the codebase is a bit unstable right now. So, if you are a Londoner, please do use it and tell me what you think, either on the comments below or on Twitter. @ me, don’t @ the bot – it will think it’s a request for a bus service and get confused. :) All suggestions are welcome.

(And now, some tech stuff for the more interested)

The bot is a Python script, run every minute via a cronjob. It’s quite short – 350 lines including comments for the main bit. As well as the live data API, the service also uses two databases officially provided by TfL’s syndication service for free; one is of all the routes, and one for all the bus stop locations. I converted these from CSV format to sqlite so the bot can make SQL queries on the data. TfL use OS Easting and Northing locations for the bus stops, so I have to convert the GPS longitude and latitude; I am indebted to Chris Veness and his lat/lng to OS conversion script, which I translated from JavaScript to Python; I am also now much more educated on subtleties like the difference between OSGB36 and WGS84. Finally, I use the Tweepy library to receive and send the Tweets, which is really rather excellent and saves a lot of faff. Finally, the whole project would not be possible without the ideals of open data and open source software behind it, so if you’ve written even a single line of free software, then thank you as well.

Although this was not solely due to what came out of the latest Facebook f8* conference, it probably was some sort of straw that broke a proverbial camel’s back. At f8, Mark Zuckerberg announced the Facebook Timeline, a way of not just showing what you are up to right now, but your whole life as Facebook saw it, digitised and shown to all. And my reaction was along the lines of:

Fucking hell, I’m going to be spending the rest of my life tagging photographs of myself

I joined Facebook early in 2007 when they let ordinary civilians in, and at first I quite liked it. It was a cute way of tying in and aggregating one’s content, thoughts and photos, and keeping up with people I knew, or used to know. What a nice service. And for free! But over time, the fun faded. Facebook kept on quietly changing privacy settings and made a landgrab for copyright of uploaded photos (later rescinded).

So, I harrumphed, tightened my privacy (a tedious task), removed a lot of personal info and content (photos, imported blog posts) and despite my misgivings, carried on with a stripped-down profile to keep in touch with friends. But as Facebook matured, and my profile accrued information over time, another unwelcome feature came about.

The practice of “Friending” someone just because you met them at a party, or went to school ten years ago with them, or you work with them, seemed a good idea at the time; it’s nice, who doesn’t want more friends? Even if they are just Facebook friends. But these are people I do not see every day, for whatever reason; as sad as that may be, over time those social ties would normally fade. C’est la vie.

But Facebook ossifies these previously ephemeral social ties; they are there forever, reminding us of the past. Whereas before we would be able to let these ties fade passively, with them laid now we have to actively “unfriend” people we no longer associate with. That’s not very nice, is it – after all, isn’t the opposite of a friend an enemy? So out of politeness, we accumulate these ossified ties, even after we change jobs, cities, relationships, as a form of digital clutter.

This was as bad as it got, until now. While social ties lingered, other content on Facebook would gradually drop off your timeline and fade away. Indeed, as online archiving extraordinaire Jason Scott observed in an excoriating critique of Facebook:

So asking me about the archiving-ness or containering or long-term prospect of Facebook for anything, the answer is: none. None. Not a whit or a jot or a tiddle. It is like an ever-burning fire of our memories, gleefully growing as we toss endless amounts of information and self and knowledge into it, only to have it added to columns of advertiser-related facts we do not see and do not control and do not understand.

Be careful what you wish for. Now our Facebook profiles will have everything we ever have, dished up by default (and while Facebook’s UI has got easier to customise recently, I bet the default will still be everything). Now it’s impossible to escape your past. Everything you have ever done that has been digitally logged by you, or your friends, can now be potentially dished up as your very own digital This Is Your Life. There is, on Facebook, a photograph of me in my early twenties, passed out after drinking too much tequila on Mexican Independence Day (any excuse, my younger self would say). That’d be on my Timeline by default, no doubt.

But it’s not because of embarrassing photos that I’m off Facebook (far more cringeworthy ones exist, thankfully on analogue prints). It’s the sense that Facebook is very much about the past. The people you have known. The relationships you were in. The things you have done. And these hang around your neck and tie you down.

Whereas what’s really exciting about the web is the things you are going to do. The new fact you’re going to find out idly browsing Wikipedia. The amazing people you meet thanks to you sharing a joke on Twitter. The inspiring blog post you’ll find via Delicious. The silly lolcat you’ll find on Reddit. Facebook isn’t offering anything what makes the Internet fun, and it’s taken this change to make me realise.

With Timeline, we’re opening ourselves up with an ever-growing obsession with the past. A quote I saw last night was “We’re gonna need architectures for forgetting”. Poetic as that line is, that’s a cure when prevention might be better – for me, in any case.

I must stress that this is not to say Facebook is bad, or Timeline is going to be a failure. Plenty of people are happy to have ossified social ties – if you are in a small, close-knit social network that is relatively static, I can see it why is a boon. Timeline will be fantastic for you, if you have been on Facebook your entire adult life, and all that data is there and well-curated (which it will be, if you have been on Facebook your entire adult life). But it’s not for me; it’s not interesting to me as a user, any more. So I’m out. Bye, Facebook.^†

* Named for Fate, the all-knowing computer in V for Vendetta, right?
† Although I’m still keeping the Facebook Like button at the bottom, just for kicks and sheer hypocrisy ;)

The episode started in the Republic of the Congo (indeed, with some material lifted & extended on from his piece It Felt Like A Kiss), and looked the near-unimaginable scale of slaughter in Congo/Zaire and neighbouring Rwanda, and the role of Western interference in the region: the Belgians’ attempts to install the Tutsis in Rwanda as political ruling class, the CIA’s anointing of Mobutu Sese Seko as leader of Zaire as a bulwark against communism, mining companies’ bloody landgrab for the mineral columbite-tantalite (used in the manufacture of chips in electronic devices such as the Playstation) in the modern Congo, and even the naturalist Dian Fossey‘s ongoing feud with Rwandans as she tried protect gorillas in the rainforest. Little to do with machines, loving or not.

The other story was more conventional Curtis fare: charting the lives of geneticists Bill Hamilton and George Price; Hamilton came up with a theory of gene-centric evolution, which Price took on further, formulating a model of altruism as a means of gene propagation. And why stop at altruism? Take it to its logical conclusion, and all manners of human behaviour can be described as evolutionary techniques and no more, and we end up being, er, gene-propogating machines. This is on much more familiar territory – the scientific establishment reducing humanity to a mere aggregation self-reproducing automata.

But unlike his usual form, Curtis didn’t attempt to draw causal attempts (the kind that usually infuriate) between the two stories. In fact, apart from Hamilton’s death in the Congo, there didn’t seem to be any link at all. Instead as the episode unfolded it focused more and more on the ongoing slaughter in central Africa, spurred on by the West’s demand for Africa’s mineral wealth. When Rwanda massacres happened in the 1990s, the developed world’s militaries stood by, while its NGOs (despite their good intentions) were powerless to stop the fighting spread to refugee camps. Curtis played a series of increasingly distressing images of atrocities with a backdrop of incongruent music.

And you begin to realise Adam Curtis isn’t even trying to make a connection, and this isn’t even a documentary. Curtis is provoking you into feeling uncomfortable, into dwelling on the loss of control, disillusionment even, with modern liberal Western society, using the history of West’s interaction with Africa – colonialism, decolonialism and the bloody end of mass capitalism – as emotional bait. Your grandparents’ generation subjugated them, your parents’ generation unleashed anarchy upon them, and now you’re fuelling the slaughter and chaos they created when you buy your Playstation. This is not a documentary designed to educate, but an art project designed to provoke – riffing off the aforementioned It Felt Like A Kiss rather than The Power of Nightmares.

And when you’re done feeling uncomfortable, Curtis gives the punchline to the film: this is why you feel so attached to bury yourself in the embrace of the machines, because deep down you no longer have faith in humanity to solve its own problems or its own inhumanity.

This is a leap of logic, a provocative one, a sign of desperation, even, but an intriguing change in style as well. Perhaps Curtis thought this the best way to make his point, perhaps he couldn’t think of any others. It’s certainly a departure from his usual form: of a triptych of tightly-wound, interconnected episodes exploring the concept. And to be honest – it didn’t work; it just made the piece look disjointed and the two storylines out of place with each other. It’s not his style – Curtis is not a polemicist – his flat, laconic commentary and his refusal to appear on camera make it impossible for him polemicise – and to use a tactic of deliberate provocation is as mechanistic a view of human beings as the very philosophy he is attacking.

And yet despite the flaws and non-sequiturs of the main story, I can’t find it easy to shake off the moral of the other tale in the episode: the demise of Price and Hamilton. The originators of the selfish gene theory were ultimately wracked by the implications of their own research: George Price’s fixation on biological determinism led to his descent into self-enforced poverty and Christian piety as a means of escaping it, and ultimately suicide when it offered him no salvation. Bill Hamilton swayed the other way, growing mistrustful of modern medicine as it was an obstruction to natural selection, leading to a belief HIV was a byproduct of vaccination programmes in the Congo, where he ended up dying. Beneath the provocation, Curtis’s underlying point, warning even, seems to be: this machinist view will drive you mad too, one day.

Coming up next: a review of episode two. Yes, I know that’s the wrong order. Consider it a homage to the man’s style…

Four years on, we have his new series, ALL WATCHED OVER BY MACHINES OF LOVING GRACE, and the chance to see whether The Trap was an aberration from true form. Like its predecessor, Curtis delves into the technical not just the historical. His basic thesis (I’m summing up) is as follows: the selfish Objectivist philosophy of Ayn Rand inspired a generation of Silicon Valley geeks to create computer systems with the aim of removing the shackles of government to create a utopia of free individuals; these same system are then used by the creators of the New Economy (led by another Rand acolyte, Alan Greenspan) to create a new economic miracle controlled by the banks, Goldman Sachs and the Federal Reserve. But instead of creating a utopia they created chaos – the machines they had such faith in failed, and despite producing an economic crisis in Asia in 1997, we kept faith in them to create an even larger worldwide crisis a decade later, from which we can see no way out of.

The problem with this is that it’s seeing too many links where there aren’t any. Not every Silicon Valley company was inspired by Rand (Curtis named one or two examples at best, neither of who leading lights on the scene), and the area’s philosophy owes arguably more to the countercultural movement and political climate of California than Rand’s self-indulgent miserabilism. It’s certainly a long way away from the conservative, East Coast, market fundamentalist philosophy of Greenspan, Robert Rubin and indeed the entire neoconservative/Chicago School generation of politicians, economists and bankers who eventually assumed political control. Cyberlibertarianism envisions a future of individuals networked together, free of hierarchy or even the state; market fundamentalism celebrates harnessing the aggregate of individuals’ behaviour for greater prosperity and stability. In short, one coast’s philosophy created John Perry Barlow, the other Alan Greenspan.

Curtis’s other flaw is to confuse “machines” with what machines actually run. A computer is just a unit for processing numbers in any number of ways. They are just boxes, glorified calculators. It’s the software we run on them that makes them do “evil” and this software is made by human beings. Having spent so long talking about a generation brainwashed by Rand, Curtis now attributes all the evils to the machines. But who programmed them? Who first thought of using them for automated trading, just-in-time manufacturing, supply chain management and all the things that are now taken for granted in the New Economy? For a storyteller who loves to peel apart the unknown and the people behind history, Curtis instead frustratingly wastes his time on peripheral figures such as Ayn Rand (who died a recluse in 1982) and Bill Clinton (distracted by the Lewinsky affair and powerless to stop the SE Asia crisis), rather than the people who built and shaped the information economy.

The result is a mess, with Curtis making oversimplified and hurried connections between various subplots. But that doesn’t mean there isn’t a story to be told amongst this clutch of different tales. How did a bunch of so-called geeks and slackers, growing from the midst of the counterculture, create a multibillion dollar paragon of capitalism? How did the conservative, sleepy institutions of Wall Street become seduced by the wonders of technology and grow hypertrophically on computer models, automated trading and complex financial instruments? In short, how did Barlow and Greenspan’s generations become allies, intertwined and taking on each other’s aspects and practices? And finally, how have we become so dependent on these systems, making them become so ubiquitous and invisible, that we didn’t notice things were going badly wrong until it was too late?

If you think this sounds familiar, it’s because Curtis used this intertwined-dichotomy style of filmmaking so well with The Power of Nightmares: the story of how the ideological descendants of Leo Strauss and Sayyid Qutb ended up as putative enemies, yet neither could live without the other, and both were grounded in the same similar grievances with individualism and liberalism. It’s odd that Curtis was able to portray the balance of similarities and differences in that film, yet with AWOBMOLG he struggles to make sense of it all, and ends up merely telling the “what” rather than the “how”, giving us numerous red herrings in the process.

It’s easy (but patronising) to say that’s because “technology is hard” and it’s difficult to comprehend it and history together rather than history alone. But Curtis is not a stupid man. It’s perhaps more charitable to say it’s easy that when it comes to relatively-uncharted history of the information age, there is simple so much more information and so many possible narratives, it’s easier to see pattens where there are none than not. But, this was just part one, and maybe parts two and three are better, and a lot more coherent.

In the meantime let this not detract from Curtis’s earlier works – if you haven’t seen them, The Power of Nightmares and Century of the Self are both available from archive.org. His KABUL: CITY NUMBER ONE blog is a collection of blogging and archive clips about the Afghan capital, and well worth reading. And I retain a soft spot for It Felt Like A Kiss, an avant-garde experimental attempt at storytelling based on 20th century history commissioned by the Manchester International Festival. AWOBMOLG was disappointing but don’t let it put you off entirely.

Another week goes by, another scrape on Twitter. This time slightly less interesting than my stumbling on the bin Laden liveblogger, but still intriguing, as I have never before seen a tech company edit its Terms of Service live in front of my eyes, like a Wikipedia entry, in response to a Twitter storm.

Twitpic is one of the leading Twitter image hosting services out there – about four million daily users, the site the famous photo of the plane landing in the Hudson River first appeared on, and the default image service for the Twitter iPhone and desktop clients. It has become big business – it brings in $1.5m in ad revenue a year.

If you run an image hosting service, you have to be careful in how you treat users’ copyright. Your users (usually) own the copyright to the photos they upload, but the service will need some form of non-exclusive royalty-free licence to legally host it on its servers. This licence is included deep in the terms of service. So far, so dull. However on May 4th, Twitpic’s terms of service changed, specifically the copyright section. The copy I have for reference is from Google’s cache and is mirrored here and it’s the first four paragraphs of the copyright section that bear most interest:

By uploading content to Twitpic you give Twitpic permission to use or distribute your content on Twitpic.com or affiliated sites.

You may not grant permission to photographic agencies, photographic libraries, media organizations, news organizations, entertainment organizations, media libraries, or media agencies to retrieve from Twitpic for distribution, license, or any other use, content you have uploaded to Twitpic.

All content uploaded to Twitpic is copyright the respective owners. If you publish content uploaded to Twitpic on the web for personal and noncommercial purposes you are required to link back to the original content page on Twitpic and attribute credit to Twitpic as the source where you have taken the content. For example a Twitter “retweet” is acceptable provided the original content link on Twitpic is what is retweeted. It is not acceptable to copy or save another user’s content from Twitpic and upload to other sites for redistribution and dissemination.

To publish content for any commercial purpose or for distribution beyond the acceptable Twitter “retweet” which links back to the original content page on Twitpic, whether online, in print publication, television, or any other format, you are required to obtain permission from Twitpic in advance of said usage and attribute credit to Twitpic as the source where you have obtained the content. No user may grant a third party permission to copy or save content that has been uploaded to Twitpic.

The first part highlighted is a clause seemingly denying anyone who uploads a picture to Twitpic the media exploitation rights for that picture; it specifically targets those businesses who might want to pay for it. The second is a more vaguely-worded catch-all clause that, in the most draconian interpretation, could deny a user from uploading their own pictures to other hosting services like Flickr.

For comparison, this was the equivalent section from the terms of service in May 2010, which is the most recent copy held on archive.org – sadly I have no more recent copy to compare with:

By uploading your photos to Twitpic you give Twitpic permission to use or distribute your photos on Twitpic.com or affiliated sites

All images uploaded are copyright © their respective owners

[Note: I am not a lawyer, and this is just a lay reading of the situation. But it will become clear, I hope, that these passages are at the centre of what went on in this kerfuffle]

Although these changes were made on the 4th, having done a little detective work on Google Realtime, it seems no-one picked up on them for six days. A single Tweet by @JMRooker on the 5th noted they had been updated, but not on what had changed. It wasn’t until this article (in German) appeared in in Der Spiegel‘s tech section that noted the change this afternoon. The first Tweet I have found was by Beate Clever in German at 15:54 UK time; at 16:49 it was Tweeted in English by Oliver Reichenstein.

Although it took six days for the news to get out, once it did it spread very quickly. Oliver’s Tweet was spotted by my We Are Social colleague Hannah, who retweeted it herself at 17:59, and I promptly followed at 18:05. By this point the story had legs – though I was by no means the only person talking about it, my Tweets on the subject got picked up and retweeted by some excellent & influential Twitter people such as Zoe Margolis and Tom Coates.

You could be easily convinced this was just another angry Twitter mob. But those who responded shared some interesting points of view. Was I being unfair on Twitpic? Did the terms only apply to the version they hosted? But if so, what makes that version different from the original copy you took? Did it only proscribe you from sending the Twitpic to a media agency, and if so would emailing a separate copy of the image as an attachment be just fine? Or was it just a protection from unfair infringement by media organisations? With the new terms of service as vaguely worded as they were, it was subject to various interpretations as the buzz spread.

The specific mention of picture agencies coincided with yesterday’s news that Twitpic had signed a deal with the photo agency WENN to represent celebrity pictures posted on the service (what Der Spiegel picked up on). So this was perhaps a foray for Twitpic to become a citizen journalist version of the PA, providing free hosting in exchange for the right to licence the rights to picture agencies. The idea has been mooted before. In theory, the next time a plane lands in the Hudson River, Twitpic’s ToS would allow it to sell the rights to witness photos uploaded to the service, and prevent the photographer from seeking those rights herself, if the company so chose. Whether Twitpic seriously thought of this as a future business model, or was just enabling these terms now in case they would come in handy in the future, we don’t know, and I am not saying one way or the other.

Finally, apart from legalities, is what they’re laying out morally fair? The new ToS brought a fair bit of opprobium (not least from myself). But, do remember, with free image hosting, you aren’t paying, and although they are getting cheaper, bandwidth and scaling up a service do cost. Online display advertising is just one business model and has increasingly tight margins, so Twitpic may be in the early stages of exploring alternatives. As with any service offered online for free – always caveat emptor.

Twitpic managed to backtrack very quickly. At 18:45, the support team hurriedly tweeted back to me (and then others in the story, such as Oliver) the same message, stating:

They were true to their word; suddenly whole chunks of the text started to disappear from the Terms of Service page as it was being edited; I managed to take notice and livetweeted the progress. Round about 19:05 UK time, out went the second paragraph about photographic agencies entirely. At the start appeared an entirely new paragraph affirming copyright holders’ rights to their work. The paragraph on non-commercial reuse was cut, save for the final sentence (“It is not acceptable…”), which was merged with the new first paragraph. By 19:52 the final edit had been made: from the fourth paragraph, the third party sentence was cut out entirely, and the start was reworded to affirm it applied to reproducing other users’ content, and did not cover your own. In the updated Terms of Service, the equivalent text to the above now reads:

All content uploaded to Twitpic is copyright the respective owners. The owners retain full rights to distribute their own work without prior consent from Twitpic. It is not acceptable to copy or save another user’s content from Twitpic and upload to other sites for redistribution and dissemination.

By uploading content to Twitpic you give Twitpic permission to use or distribute your content on Twitpic.com or affiliated sites.

To publish another Twitpic user’s content for any commercial purpose or for distribution beyond the acceptable Twitter “retweet” which links back to the original user’s content page on Twitpic, whether online, in print publication, television, or any other format, you are required to obtain permission from Twitpic in advance of said usage and attribute credit to Twitpic as the source where you have obtained the content.

Those changes might be a bit much to get your head around, so to make it easier to appreciate how big they are, I have created a side-by-side comparison of the two wordings – old is on the left, new on the right – and created a diff of them (the changed bits in blue). Feel free to click through to a full-res version.

You can see for yourself that the changes are quite extensive, and the terms are now vastly stronger in their affirmation of uploaders’ rights. A mere clarification this is not, in particular the second sentence of the first paragraph – “The owners retain full rights to distribute their own work without prior consent from Twitpic” – is in marked contrast to the original wording.

Twitpic have published a blog post called “Your content, your copyrights“, which states their reasoning for changing the Terms of Service:

As we’ve grown, Twitpic has been a tool for the spread of breaking news and events. Since then we’ve seen this content being taken without permission and misused. We’ve partnered with organizations to help us combat this and to distribute newsworthy content in the appropriate manner. This has been done to protect your content from organizations who have in the past taken content without permission. As recently as last month, a Twitpic user uploaded newsworthy images of an incident on a plane, and many commercial entities took the image from Twitpic and used it without the user’s permission.

It’s great that a hosting service is explicitly protecting its users from exploitation and unlicenced copyright infringement. But the original draft of these updated terms of service made it clear that it was about more than just preventing unscrupulous news organisations from misusing photos. The now-removed sections were clearly about preventing users from selling their rights to their own uploads to third parties, not protection from theft. And unlike the apologetic blog post on May 10th, these initial changes to the ToS were not publicised to the wider community when they went up on May 4th; as I have detailed above, it took six days before anybody actually noticed – and by the way, all the credit should go to the tech team at Der Spiegel for spotting it. How Twitpic went about this change is not how a tech company should publicise changes to its users; whatever their motives were for updating their ToS or whatever plans they have for their business model, at the very least this was a major failure in communication.

Twitpic do deserve some plaudits for reacting quickly to the situation, answering those of us who questioned it on Twitter, and updating the Terms of Service to something more acceptable in ludicrously quick time and manner. I say more acceptable. They do still retain a licence to distribute your content as long as it is done in connection with their business, and that business model could well change from being an ad-supported image host in future. There’s nothing stopping you from exploiting the rights to your image, but they have those rights too. You may be fine with that, in which case carry on, or you may not feel entirely comfortable, in which case you may still want to choose somewhere else to host your images. I’ll repeat what I said above, when it comes to free stuff, caveat emptor, especially if you reckon you stand a chance of one day being the next person to snap a plane in the Hudson.

Update (18/05): Tom has received a letter confirming Twitpic’s intentions to “exclusively” resell photos through the WENN picture agency. There is still no announcement of this on Twitpic’s own blog or in the terms of service. So, what gives?

The television news we were watching had nothing new to show by now, and was resorting to reruns of President Obama’s address earlier. So, with my curiosity piqued, I started looking up to see if there had been any coverage of helicopters in Abbottabad earlier that day. Googling around normally found the odd news report about a possible training accident, but very little of substance or interest. So I turned to searching Twitter, specifically with Google Realtime, which allows you to exclude Tweets from before or after a certain time of day. This was important, as once President Obama had disclosed the location, Twitter exploded with mentions of it and it became impossible for ordinary Twitter search to cope.

With anything after 11pm Eastern Time excluded, I was able to find Tweets by a guy called Sohaib Athar, or @ReallyVirtual. Once I clicked through to his timeline, I found out he had actually liveblogged the entire raid, unaware that it was America seeking its public enemy number one. At 12.38am, I tweeted, and Maha tweeted too:

And then at 12.41, three minutes later, Sohaib tweeted the defining moment of the story:

(Tip: Twitter annoyingly displays Tweets’ times in a “x minutes/hours/days ago” format, but if ever you want to check the exact timing of a Tweet, hover your mouse over the that bit and a tooltip will give you the exact date & time, in your timezone)

By the next morning, Sohaib was one of the most famous Twitterers around, being interviewed on television and getting mentioned in most mainstream media outlets. His follower count shot up from 750 to just over 100,000 as of today.

Steve Myers of the Poynter Institute got interested in how the story spread and did some investigating (including talking to me and Maha), producing a phenomenal forensic blog post – and from his investigation it appears that mine & Maha’s Tweets were one of the first ones to mention him and may have broken the story.

Caveat: I say may – correlation does not imply causation. I looked on Google Realtime for earlier Tweets from anyone linking to his account pointing and highlighting his liveblogging, and could not find any, but that doesn’t mean there weren’t any. Nor am I trying to take too much credit for breaking the story – had I not tweeted about him, someone else would have found him sooner or later – the tools were there, and reasonably well known in the trade (and if you work in the media and don’t know them, then for God’s sake learn them).

Steve’s piece is a great bit of detective work and social network theory in one, and I’d like to pick up on a few points he made. Firstly, he states “the number of followers doesn’t matter as much as who those followers are” – this is a really interesting one and worth elaborating. I have about 3,300 followers on Twitter, but most of those are UK-based and would have been fast asleep. Maha has 1,300 followers, but she is a journalist based in the US, specialising in amongst other things, Pakistan. Her following, while smaller, is full of journalists, policy people and those with an interest in Pakistan and the wider region; these would be the exact kind of people to pick up on the significance straight away.

The numbers then don’t always count. But what definitely does count is the story. Steve picks up on the role I played bridging different social networks (in a paragraph that makes me feel odd, being referred to by my surname…):

Applegate was a bridge too, in a slightly different way. He added essential information that resonated with people and spurred them to pass it on.

I didn’t regard myself as a bridge at the time. I just thought, and tweeted: “Wow”. But then as it unfolded more it became clear that the unwitting Tweeting was a central factor in the story. Abbottabad is a relatively quiet town, populated by retired generals and known for its schools and universities (not to mention their military academy). Sohaib himself had moved there to get away from the much more dangerous and turbulent Lahore to find quieter climes – his Twitter bio states he is “an IT consultant taking a break from the rat-race by hiding in the mountains with his laptops.”

Given a popular narrative of Bin Laden hiding in caves and the like, to find out he was living in a mansion somewhere so quiet, so genteel and so near to the heart of the establishment came as a surprise. The key thing that made Sohaib’s liveblogging from earlier in the day so compelling was that it was completely unwitting, mirroring our own disbelief that Bin Laden had been quietly residing in the Pakistani equivalent of Tunbridge Wells all these years, without any of us knowing. The story chimed perfectly with our own emotions. And because the story had been unwitting, it was also candid and honest, cutting through the hype and speculation that the 24-hour news stations were resorting to.

Finally, the whole episode shows how transformative Twitter can be. As the story matured and his fame rose, Sohaib took on the role of citizen journalist, becoming a correspondent of sorts (not many other residents of Abbottabad are on Twitter, he remarked, it’s mostly Facebook). He conducted interviews on television, and ventured out into town to take photographs and report back on the mood in the town.

This is a far cry from the cynical caricature of Twitter as an echo chamber – a place where nothing new is said and everything is relentlessly retweeted. As the story progressed, Sohaib came to the wider community’s attention and it in turned shaped his role in the affair. His relationship with Twitter evolved – it went from being a place to remark on the events that had taken place, to realising their significance, to realising his own significance, and then finally embracing it with intrepidness, intelligence and good humour. I might have been one small factor that sparked the process off, but I definitely can’t take any credit for the phenomenon he has become – that’s entirely to his own credit, and something that we should celebrate.

Update (05/05): Maha has also blogged about the events, if you’re interested.

The political subtext of government data
One of the reasons we can be more serious these days is that the years of relentless campaigning have paid off, and we are now getting more and more open data from the government, and other sources, to mash up, with a variety of results such as traffic injury maps, finding postboxes or visualising spending cuts.

This is, by and large, fab – getting the full potential out of the data that has been gathered by the government at public expense by letting the public explore it. It’s even possible to envision how this data can be used to disrupt or even disprove party political beliefs and theories by exposing them to cold, hard data. But a deluge of data does not mean the end of theory (as Chris Anderson has expounded). Data is not some cut-and-dried artefact free of politics or prejudices.

Hadley Beeman‘s talk on the challenges facing those playing with such data. She explained often there are things such as reference numbers, acronyms – a whole unspoken culture behind the data – which can get stripped out when presented to the wider public. And this got me thinking about subtext behind datasets; what are the unspoken assumptions being made in their collection, or the process behind the design of the system that has collected them?

For example, lets take performance data from schools; usually visualised as a league table, they become the focus of obsession by parents. The league tables have however become the focus of much ire from within the education profession, BERG have attempted to make more of the data with their Schooloscope project – which looks lovely and more user-friendly than columns of figures in newspapers, but misses the point – the league tables aren’t demonised because of their format, but because they may not accurately represent the performance of a school – they may ignore social disadvantages a school’s intake may suffer from, ignore extra-curricular activity or that “soft” subjects are given the same prominence as “hard” subjects. Which of these factors you think really matters will largely be down to your political beliefs, and conversely, the decisions that led to this data being recorded and the way it was assessed, broken down and analysed will also be politically influenced.

There is a feedback loop as well – recording this data in a particular way can end up affecting the very thing we’re trying to assess. After years of being incentivised to perform better in league tables, schools are now accused by some of being little more than coaching centres for children to pass exams than providing them with a full and rich education to prepare them for life. This is not a universally-agreed fact either, but an opinion shaped and refracted by the critic’s political beliefs; even if you agree it’s happening, you may disagree on whether it’s a good or bad thing. In short, the whole process of collecting data – supposedly simple, neutral and objective – opens a can of political worms and can create polarised debate. Simply opening up data and casting many eyes over it is not going to make the controversies about these data go away. And in fact, by doing so without questioning the subtext, we can end up unwittingly complying with the social and political aims of those who collect it.

This might sound a bit paranoid, and wanky, and so I’ll stress that this should not dampen enthusiasm for doing more with out data. The data being opened up (not just by the government, but by the BBC, the Guardian and many other providers) has so many potential uses and ways of enriching us socially. But at the same time we should always be questioning the provenance of data, think about the decisions that had to be made in structuring that data, and asking not just about the data we have got, but what useful data might be missing.

Context, failure and hindsight
Another thing that Hadley mentioned, and worth considering is that occasionally civil servants make mistakes collating data. At the moment this can be difficult to annotate, to explain where a mistake has happened and how it was made – and this is something we need to encourage, for else how will organisations learn? But to demand this we also need to possess a degree of tolerance ourselves. Hindsight is 20/20, and it’s easy to apportion blame quickly (especially in a post-Twitter age of instant reaction), but doing so may end up being counterproductive. Indeed, a culture of fear may already encouraging civil servants and politicians to stop recording controversial meetings or opinions for fear of being found out later with an FoI request (as anecdotes from Ireland have hinted at). Less apoplectic rage and a greater tolerance of sharing stories of failure are needed if we want free thought and debate inside our governments – indeed, that’s a lesson that has lots of applications outside of government data as well.

Futureproofing and archiving
Bill Thompson gave a stirring talk on the need for archiving our analogue past digitally, before we become so detached from analogue that we don’t think any of it is worth saving. Bill took the pessimistic view that our kids might not archive our analogue stuff, which I think is a little unfair on them, but if that warning spurs us on to get it done then the ends justify the means, I guess.

The obsession with archiving now has struck me as somewhat odd – we live in era where storage space is near-infinitely abundant and yet we are more worried about losing our culture than any other age in history. Did the scribes of the Lindisfarne Gospels factor in the possibility their work would still be around 1,300 years in the future? Even once a cultural artefact has become deemed a classic, preservation has often not been on the minds of those in charge of them – Michelangelo’s David was left outside exposed to the elements for centuries. Even attempts to preserve works, such as with Da Vinci’s The Last Supper, or Stonehenge, can involve damaging or radically altering the original so it is no longer the same as what it was, leaving us potentially with a Ship of Theseus rather than a “genuine” cultural artefact. Then again, in an age where people make money selling fake versions of forgeries, maybe that doesn’t matter so much?

As information has become less scarce (we now apparently produce more bits in two days than we did in all human history up to 2003), paradoxically we’ve become increasingly obsessed with preserving it. Maybe it has something to do with the volatility of our storage – all it takes is your hard disk to be corrupted and you could lose years of your work. Or the effect of the internet on giving us information at our fingertips means we’re now capable of knowing what we would lose if these archives disappeared. Or maybe it’s just hindsight and a selective memory – we lament all those thoughtlessly-wiped episodes of Doctor Who, and are now much more sensitive to data loss, but we’re not so fussed about all the editions of The Cliff Richard Show that got deleted too.

Or maybe it’s because digital archiving implies, with the limitless copying it allows, perfection and immortality. Once a cultural artefact is scanned, ripped and uploaded, then we can make as many copies of the digital version as we like, and that digital version will be perfect – so we don’t have to worry about losing or mutilating the original. But then that relies on an awful lot of assumptions. How long will the hard drives or DVDs we store them on stay true, and will we always have device drivers for them? Will the HTTP protocol, or the JPEG compression algorithm exist in 100 years time? Will we even think of computer data as something stored on machines as 1s and 0s by then?

There is a warning from the not-too-distant past. The BBC Domesday Project of 1986 was an attempt to digitally record Britain on laserdisc, like the original Domesday Book of 1086, yet within 15 years the discs were almost unreadable due to a lack of suitable equipment (thankfully, geeks have now made sure the format lives on). A working group I once attended at Cambridge discussed points like this when talking about approaches to digitising the university’s library; the magnitude of time one person was talking about was in the tens of thousands of years. It was exciting stuff – rarely do we ever consider our future as long as that – but also sobering.

Fortunately, the BBC are more forward-thinking than some other organisations, and the lessons of Domesday were learned a long time ago; judging from their posts about archiving, futureproofing digital formats is foremost in their thinking. Digital doesn’t necessarily entail persistent. As an aside, this can act as a reassurance to those worried that youthful digital transgressions could ruin their future lives. Most of the stuff I’ve created online up until 2003 (when I started this blog and properly archived stuff) has now disappeared into the aether, maybe only accessible through dipping into archive.org; whatever youthful transgressions there may have been are now gone. A lot of data can and does get lost over time.

So yes, let’s archive as much of our analogue past while we still can, we will be the richer for it culturally, but let’s not think it necessarily means it will live forever. And while we’re at it, we should become more comfortable with the notion that it’s okay if we lose some stuff from time to time – it’s a fact of life, and if we get too obsessed with preserving everything, we’ll never have time to make anything new.

Excellent stuff to look out for
In short: Ben Goldacre‘s launching a project to keep track of abandoned or never-published medical trials. Keep also an eye out for Rob McKinnon’s Whoslobbying.com as well. The guys at Young Rewired State showed that despite the relatively poor provision of teaching code in schools, there are some great young talented enthusiastic hackers coming up and making things like this. I missed the talk about Frontline SMS but really like the idea – not everyone has a fancy smartphone after all (see also Terence’s excellent talk on designing for all phones). Finally, I will probably be playing a bit with Scraperwiki and the datasets on data.gov.uk, amongst other things…

Last week I took part in one of the most fun and interesting blogger events I’ve ever been invited to. Nom Nom Nom, or “The Bloggers’ Masterchef’, is a cook-off organised in aid of Action Against Hunger by the ever-enthusiastic and energetic Annie Mole (of Going Underground fame), held at the fantastic Cookery School on Little Portland Street in central London.

I felt a bit daunted before the event kicked off. I’m not a food blogger – I write about tech and politics and take the piss out of the Daily Mail; I’m someone who very much enjoys their cooking, but to be in the kitchen with “proper” food bloggers? I was intimidated. Luckily, my cooking partner (and flatmate) Tom (aka @flashboy) has done one of these before, and my nerves were (just about) calmed when he said it wasn’t as competitive as I feared.

Like all good geeks, I made sure I read up and practiced beforehand, eventually settling on three dishes that were summery. Tagliata, seared Italian beef with rocket & tarragon to start; sea bass baked on vine tomatoes with spinach, pine nuts and raisins for the main, and an English summer berry trifle as our dessert (and also our compulsory cold dish). We can’t claim originality – the starter & main came from Tom Norrington-Davies’ Eagle Cookbook, the trifle from Nigel Slater’s Appetite.

The blessing with nearly all of what we cooked is that we could get UK-based ingredients in season – the beef English, the sea bass from Anglesey, the fruit & vegetables from local farmers’ market – it was only the small things like the olive oil and raisins that would have to come from further afield. Furthermore, all the dishes were relatively easy to make and not too daunting, especially in a high-pressure environment.

The tagliata went like a dream – a really nice cut of Hereford sirloin, seared on grillpan and then thinly sliced. Tarragon is an odd choice of herb to go with beef, but there was something about the aniseediness which works well with the rocket. The recipe we had also called for new potatoes – in retrospect though they were probably a distraction from the dish and didn’t add much.

The fish was perhaps the simplest of the three dishes to cook, just season well, slash the flesh open to help it cook a bit quicker, and lay down on a bed of juicy sweet tomatoes & sliced garlic. I might have overdone it with sloshing the white wine on, which ended up making the toms being a bit soggy, but it still tasted fantastic. You might worry sea bass is a bit delicate to be overburdened by tomatoes, but it actually works out fine.

Tom took charge of the dessert – alas in order to conform with the no cooking rule we had to use ready-made custard. A chance encounter in the newsagents led us to find some sherbert flying saucers, so he adorned each of the sundae glasses of trifle with them, which ended up as a really nice quirky little touch.

The upside of all of our dishes was that they didn’t require that much preparation. The downside is that they didn’t take much time to cook either, so after a lull in the middle after all the prep, the final few minutes were a real stress. We didn’t have a big enough pan for the spinach, so we had to do it in batches, without ruining by burning the pine nuts (something I was very careful not to do). By the time it came to plating up I was in a rush, so it wasn’t as neat as it could have been.

We weren’t expecting to win, yet… well it turned out we were right, ‘cos we didn’t. But I was happy with what we cooked – especially when it was clear there were some genuinely talented cooks in the competition. Nevertheless we did get some lovely plaudits about our sea bass, and the trifle, including from the winners, which I’m going to to take as a top-grade compliment. Best of all, it was a real pleasure working in a proper kitchen, and with proper staff – the Cookery School’s staff were absolute angels from start to finish, tirelessly helping us with our every whim, and not minding when pressure meant there was no time for “please” and “thank you”.

It was also really good getting to know other bloggers, and indeed getting to know my own flatmate better – Tom tends to downplay his own culinary skills but at the cookup, but after that I now know he is a perfectly good kitchen lackey assistant chef in his own right as well.

Many thanks go to the Cookery School (whose kitchen really is excellent), and the many people who made it happen, including Rosalind, Annie and Chris Osburn (who took all the pictures above, and I’m very grateful for him doing so, as I had no time to take proper pics). The event’s not quite over yet – the sponsors have donated prizes to a charity raffle in aid of Action Against Hunger – go buy a ticket now to help make a difference.

Update: The Viewer’s Choice award is now up and running over at the Nom Nom Nom website – if you liked the look of what Tom & I cooked, or you just like us anyway, then place a vote for our team, Nom Nom Nom De Plume. Vote early, and vote often – you can vote once a day ;)