as days pass by: a weblog by Stuart Langridge

How to make APIs that people like

sil — Sat, 12 May 2012 00:00:00 +0000

A short presentation I gave at today's Multipack Show and Tell on "How to make APIs that people like".

The tl;dr version: your API is part of your user experience. Take your UX knowledge -- design for the user, not for yourself; think about the user journey rather than just modelling your underlying infrastructure; make it understandable and intuitive and focused on their need -- and do all the same things when designing an API.

Waiting

sil — Thu, 03 May 2012 00:00:00 +0000

Buying Happiness — Jeff Atwood

Pay now and consume later: Immediate gratification can lead you to make purchases you can't afford, or may not even truly want. Impulse buying also deprives you of the distance necessary to make reasoned decisions. It eliminates any sense of anticipation, which is a strong source of happiness. For maximum happiness, savor (maybe even prolong!) the uncertainty of deciding whether to buy, what to buy, and the time waiting for the object of your desire to arrive.

Next time Jono complains at me for spending six months agonising and analysing about a new phone or new laptop or something, I shall tell him: it is science.

Ubuntu One Files for the Nokia N9

sil — Tue, 01 May 2012 00:00:00 +0000

A first release: Ubuntu One Files for the Nokia N9. Includes short YouTube video with nice music. Still lots to do, of course.

Conference wars

sil — Wed, 25 Apr 2012 00:00:00 +0000

Just had an odd little idea.

Imagine you go off to a conference -- one of the ones where you wander around and look at stuff and talk to people in corridors and so on, not one of the ones where you sit in the same seat for the whole day and then go home. And when you pick up your lanyard and badge, there's an extra little stack of cards and you can take one, if you want. A card has on it, say, ten empty boxes with "Deaths" written above them, and it's a little bit taller than your conference badge so when you put it in the badge case it pokes up above the badge a bit. What this means is that taking one of the cards means you're playing the conference game, and other people can see that you are (because the card pokes up a bit). So, when you're walking around and you see someone else playing the game... you Roshambo them. (That's Scissors-Paper-Stone, or Rock-Paper-Scissors, for those who don't know the term "Roshambo".) The loser takes out their little card and puts an X in one of their "Deaths" boxes, in sight of the winner. Then each goes their separate ways. If all your Deaths boxes have been Xed then you're dead and no longer playing; put your card in your pocket.

No idea if this'd actually work, but I think it'd be cool. Fun little thing to be doing while you're enjoying the conference. I can imagine it would have worked well at LugRadio Live, for example.

Extra thoughts

On the back of the card, have a special move, and a number of "hour" boxes, one per hour in the conference day. A special move would be something like, say, "Dynamite" -- if your opponent plays rock, you automatically win, regardless of what you yourself played. If you use a special move to win, visibly put a tick in the corresponding hour box (so you can only use a special move once per hour, or five times in total, or something).

If it gets close to the end of the day, how do still-alive players find one another in order to do battle? Haven't quite worked this out yet.

Printing up a stack of the cards would be cool, but I suppose it's not strictly necessary -- someone can just draw themselves a card if they want.

There are a bunch of sensibleness rules -- you can't fight the same person twice in succession, for example -- in order to stop two people fighting until one of them is eliminated from the game entirely. Of course, given the rules lawyers, probably you need to stop three people standing in a triangle and fighting until one is eliminated, but that's basically covered by the generalised "don't be a dick" rule :)

It should be possible to tweak the cards for the particular conference... if you've got a zillion attendees and they all want to play, maybe you need more than ten deaths, or you get more special moves, or whatever. That seems easy enough.

Let me know if you try it out.

Decades

sil — Mon, 23 Apr 2012 00:00:00 +0000

I've been writing this blog for ten years, as of this weekend.

In that decade, I've written 510,216 words here, which is about the same length as War and Peace or Les Miserables although it's a little more focused on technology than on the impact of the Napoleonic era on Tsarist society. After poking around with a bit of maths, a few stats in the form of a bunch of small bar graphs.

A bunch of small bar graphs

Posts per year	▇▄▅▅▃▂▂▂▁▁▁
Word count per year	▇▄▃▃▃▂▂▂▁▁▁
Word count average	▃▂▂▂▂▃▂▃▃▃▇
Amount I mention Ubuntu	▁▁▁▁▁▁▁▂▄▇▄
Average number of comments	▁▅▃▃▃▄▄▇▅▆▆

That last one's interesting. It seems that the number of comments on posts has stayed relatively steady over the last ten years, which is nice. There are a few things that skew that, though:

Publishing a folder with Ubuntu One

sil — Wed, 18 Apr 2012 00:00:00 +0000

Ubuntu One has the concept of publishing a file — that is, giving a file in your personal cloud a URL so that anyone can download it, if you tell them the URL — but you can't currently do that with a whole folder of files. Since that'd be quite a useful thing to be able to do, and it's not yet supported by U1 itself, I wrote a little script to do it, u1-publish-folder (the script needs you to be on Ubuntu 12.04, which is released next week). Simply do u1-publish-folder /path/to/synced/folder and it'll give you a URL for the index for that folder; a handy way to get a bunch of files to someone without a U1 account* where they can pick and choose the ones they want (rather than sending them a zip file, or sharing a folder with them).

The way it actually works is to publish all the files in the folder, then create an HTML index file linking to all those published files, then sync and publish the index. So you get a directory listing, like Apache gives you when you browse a folder: List of files in testpublishfolder.

The thing I found nicest about this is working with SyncDaemonTool, the Python U1 controller. Combined with Twisted's defer.inlineCallbacks stuff, it makes dealing with U1 really easy:

@defer.inlineCallbacks
def dosomething():
    sd = SyncDaemonTool()
    metadata = yield sd.get_metadata("/home/aquarius/Documents/somefile")
    print metadata

Normally, dealing with async stuff is hard, but it's just dead easy this way; simply throw in a yield and you can write code as if it's normally synchronous but it's actually async, so your program doesn't block while it's working and you don't need to invent a zillion "callback" functions (whether they're anonymous functions or not). I like this.

Potton in 2012, a video drivearound

sil — Mon, 09 Apr 2012 00:00:00 +0000

About six months ago I saw a YouTube video of Bedford, the nearest big town to where I grew up, in the mid-eighties. (It probably wasn't this one but that's fairly close.) It was fascinating; we used to go shopping in Bedford back then when I was a kid, and seeing all the shops that are no longer there was most interesting. Anyway, I mentioned this to my dad, and he said: we should do something like that for Potton, the town in which I grew up (and in which my parents still live). Today we did exactly that; since Potton is a small town, we drove down every street in town and recorded it on video.

You learn a new respect for the Google Street Car people doing this, I can tell you.

I wanted to add maps and details of which street was which in the video, and after some playing around with the OpenShot video editor* I worked out how to do just that. Each street gets a couple of seconds of "title" showing its name and also where it is on a map of Potton. The maps were made possible by OpenStreetMap, which is like Google Maps crossed with Wikipedia; it's a map of the whole world, but it's created by people and not by corporations who want money to licence it, and therefore you're allowed to use the maps to do interesting things.

Warning: technical stuff about vector graphics formats coming up. Skip this paragraph if you don't care.

What I did was get a map of Potton from OSM, export it as SVG, and then go through that map in Inkscape and mark each road with the name of that road. I was enormously pleased to see that each road on an OSM SVG is an individual path (making this possible). OSM people: if you wanted to do something really cool, then in an SVG export make each path have the name of the road that it is as its ID (rather than "path2319" or whatever). This is probably quite hard (each road has at least the path defining the street itself, a path defining the border of that street, and the text), but naming these as "roadHighStreet_main", "roadHighStreet_background", and "roadHighStreet_text" or something might be possible. Anyway, once I had an SVG with all the IDs set to be the street names, a quick Python script was enough to turn that into one SVG per road with that road highlighted in red. (You can get the SVG of Potton with all roads marked with IDs as well, if it's useful to you.) The script also creates an "overlay SVG" for each road name as well, so that I could overlay the road name onto the video so you can see at any point where you are in town.

End of technical stuff.

This was a fun little project. I mean, what you've got is a video which is nearly two hours long and is nothing but driving round a small Bedfordshire town, so it ain't gonna win any Oscars, but I'm pleased we did it. Maybe in twenty years someone will find this on YouTube and be amazed at what Potton looked like in 2012.

Potton in 2012, a video drivearound (and downloadable in webm and h.264 format)

This time, more than any other time

sil — Tue, 27 Mar 2012 00:00:00 +0000

"Here lies Edmund Blackadder. And he's bloody annoyed."

Tonight I went to see my daughter dance in a show. Also there with me was my ex-wife's mother, and the chap she married a couple of months ago. So I suppose that would make him my future-step-ex-father-in-law. For simplicity we'll call him Rowland. Anyway, I wanted to video a particular bit of the show (a bit with Niamh in, obviously) but couldn't since I was a row or two back, so Rowland did it on his phone. During the interval I said: I'd like that video, and I know Sam* would too because that particular bit wasn't in last night's show, which she saw. And Rowland said: OK, how do I get it to you?

Now, this is a modern flagship smartphone he's holding in his hand, not some ancient Sagem thing with buttons. So I said: just email it to me. How do I do that? he says, and hands me the phone. It turns out that he's got this phone in order that he can get the football results on it, and to send text messages. Never set up an email account: didn't even know you could read email on phones. He's a smart guy; just doesn't care about the technology, and who can blame him? Still, OK, I've got a video on this phone, and I want to get it to me, somehow. We've got good internet coverage. So, how do I do it? Guess away!

email it to my own email address: can't do that, because he hasn't set up his email account, and I can't ask him to do so because we're sitting in the audience of a show and he doesn't know how anyway.

text it to my phone: nope, video's too big (40MB or so)

upload it to my Ubuntu One account: before we start here, this is his phone, so I was understandably wary of signing into any of my accounts on it, but since he doesn't have any of his own accounts set up, perhaps I have no choice. So, reluctantly, I sign into Ubuntu One as myself and try uploading… and the browser crashes. Try it again, crashes again. Hm, thinks I, sounds like we've got a bug there. Sign out of U1. What else can I try?

Use some temporary file upload service: ok. How does one find one of these which isn't just for warez and porn and hooky videos? (You may have noticed that I am living xkcd.com/949/ at this point.) Bit of Googling, and I found one which looked sensible (note: the key trigger word, at least at this point in time, is "html5"; it may get you a bunch of hot air and bullshit from industry analysts, but if a site mentions it then it's probably relatively modern, at least.) Try uploading (I can't remember which one I found), and… browser crashes. So it's not an Ubuntu One problem (phew!), it's just with file uploading from this browser. Bloody fantastic. What else can I do?

bluetooth it to my phone: tried that, and my phone rejected it but I don't know why. Can you send 40MB files over Bluetooth? Maybe this is a fault in my phone. Don't know. Didn't work. Next attempt.

sign in to my email account and send it to myself: OK. There are three email clients on this phone (three email clients for fuck's fucking sake!?‽). One of them is a gmail app. I am scared of signing into my gmail account in an official gmail app in case the phone Learns My Account and I can't remove it easily. Second email app says "Your trial has expired. Please renew your subscription" (honestly? Trialware on a phone? What kind of shitty world do we live in where this is a good idea??). Third email app, I create a new email account which is my gmail account (with fear and trepidation), and try sending an email to myself with the video attached. "Email sending…", it says, followed by… nothing. No indication that it had worked or had not, except that I had no email (and that there was no way that 40MB had uploaded that fast). Tried it twice; same thing. Delete my email account from crappy email client. Next.

sign into my gmail account in official gmail app: fearful plan, this. Fortunately, it failed early enough that I didn't have to try; attempting it gives "file too big to attach". Thanks a fucking bundle, gmail. Oh no wait, no thanks at all. Try something else.

Now somewhat desperate. There is a "flickr" app here. Maybe I can upload the video to my flickr account? Flickr does videos now, I think. Sign in to flickr app with ancient Yahoo ID, video starts uploading! yay! FIFTEEN MINUTES later it gets to 100%, and... then just sits there at 100% without completing. And then it goes back to zero and starts again. It gets to 65% done a second time and then the show ended. Since my only two options were to go home or to throw Rowland's phone into a volcano and then go home, I gave him back his phone and went home.

We have failed as an industry.

Now, there are those of you reading this and thinking: that wouldn't have happened if he'd have had this phone or this app or this software or this service. Stop thinking that. This is a guy not interested in technology who bought a flagship internet device and has been appallingly let down by that. How will he be helped by you sneering and using his failure as an excuse to score political points? We're in this to help users, remember: not just the ones who think as we do, but the ones who rely on us to build things for them because they don't know what they're doing. If your response is honestly "well, he should have spent more on a phone to get something better", then I'm exceedingly disillusioned by you. "The internet brings freedom, but only to rich people"? Really? Is this what the Open Web, the technological revolution is all about? If your answer is "he needs different software"… really? Are we ever going to get past the point where saying "I have a problem" just gets you a hundred responses about how it's your fault for having picked the wrong computer? We have failed as an industry. If the best we can do is fight amongst ourselves and scratch like cats in a bag then it's hardly surprising we fail so badly. Then again, perhaps we're not in this to help people; we're in this to make money and helping is a sort of epiphenomenon. Some of us may be thinking: a company who builds a product this frustrating will surely be eliminated by the marketplace. Leaving aside for a moment how the tech crowd, normally as liberal a group of people as you might want to meet, become rabid free-market boosters when a product we like becomes popular (football teams have had a chant about this pernicious behaviour for decades now. It goes: you only sing when you're winning), think about all the people who get inadvertently screwed by this laissez-faire capitalism. Every company who gets "eliminated by the market" still sold a shedload of devices to a shedload of people, and all we have for them is to say "soz, dude, you made the wrong choice, you've been eliminated by the market, you are the weakest link, goodbye"? Can't we do better?

There's nothing we can do, though! comes the cry. What can we, innocent technical people, do to influence the movements and decisions and marketing and choices and products of some of the largest companies on earth? I don't know. But take that feeling of helplessness you have right there, and now imagine how helpless you'd feel if you didn't even understand the technology. That's how helpless our users are when we get it wrong, or when someone else gets it wrong and instead of helping we tell screwed people that it's their own fault for buying the blue phone instead of the one we recommended.

Niamh danced superbly, by the way.

Step away from the localStorage

sil — Tue, 20 Mar 2012 00:00:00 +0000

There has been much discussion (example, example, example, example) recently about how the in-browser localStorage API is terrible and must be killed with fire.

My first reaction here is step away from the bacon, son, leave the bacon out of this. The localStorage API seems to have a few quoted problems, the main ones being that it's synchronous, it does file IO, and it's got bad event support. Put against that, rather snidely it seems, is the benefit that "it's easy to use".

Did I miss a meeting? Is being easy to use suddenly a bad thing? Isn't that one of the major benefits of using Apple stuff, that it makes sense and people can get it without a PhD in computer science? Or does that only apply to users, who are presumably too busy on the sofa with a bag of Cheetos, and not to developers, who are meant to do things the hard way whenever possible?

Doing file IO and having bad event support are, certainly, issues with localStorage if you're trying to use it for any kind of really serious data storage. If you're Gmail, maybe you want something better. But you know what? Most of the things I build aren't Gmail, and most of the things you build aren't either. I don't need event support for localStorage; I just want to stash a value in it and then move on. That's all. Doing file IO… well, any storage mechanism had better be doing file IO at some point so that my data gets saved to disc!

Ah, no, say the detractors, but it's synchronous. That's bad: it blocks. This seems to boil down to two problems: it might block my tab, and it might block other people's tabs. Now, that's an issue, right enough. However, blocking my tab is something that I can think about and decide whether I'm prepared to run the risk or not. Because what you're asking for is that I replace this:

localStorage.setItem("key", "value");

with this

storeSomething = function(key, value) {
    var request = indexedDB.open("mydbname",
        "This is a description of the database.");
    request.onsuccess = function(e) {
        var v = "1.0";
        var db = e.target.result;
        if (v!= db.version) {
            var setVrequest = db.setVersion(v);
            setVrequest.onsuccess = function(e) {
                var store = db.createObjectStore(
                    "myobjectstore", 
                    {keyPath: "timeStamp"});
                var request = store.put({key: value, 
                    "timeStamp" : new Date().getTime()});
                request.onsuccess = function(e) {
                    // now carry on with what I was doing
                };
            };
        }
    };
}

and you know what? Given that choice… I believe I might at least sometimes choose that blocking my tab for a microsecond and very occasionally blocking my tab for two whole seconds is actually OK. I'm allowed! It's my tab!

Fine, I'm being a bit facetious. There are problems with localStorage, and it would be good to see them resolved. But there are alternatives other than "you must ditch it and use IndexedDB, Mr Developer", I think. What would be wrong with providing an asynchronous variant of localStorage, so I do localStorage.setItemAsync("key", "value", function() { carryon... })? Mozilla say "Implement localStorage in an asynchronous fashion in browsers – actively disregarding the spec? (this could set a dangerous precedent though)", as if the idea of the browser manufacturers working together to adjust the spec is just impossible; did I miss something there? The spec process is open to stuff being adjusted or added to; that's the point of it, right? And localStorage use in my tab blocking other people's tabs… I can't think of a nice way of saying "that's a browser issue", but it is. I feel sad having to say that, because I know that the engineers at Mozilla and other places have done superhuman work in this area and will continue to do so, but really one tab blocking causing others to block is surely something which is a problem for more than just localStorage?

There are problems with localStorage. Real ones. But I think that those calling for its complete removal are vastly underestimating the attractiveness of an API that's easy to use, and also vastly underestimating how annoying it is to be forced to use something way more complicated because it copes better with pathological cases. I'd love to hear more argument on this subject, but I don't want it to look like there's a consensus around "it must go". Someone needs to speak up for developers who just wanna get stuff done, like me...

Ubuntu Accomplishments and Ubuntu One

sil — Fri, 16 Mar 2012 00:00:00 +0000

Those of you who read Planet Ubuntu will doubtless have seen Jono Bacon talking about the Ubuntu Accomplishments system that he's been working on along with a growing community of developers and testers. That system uses Ubuntu One to communicate your accomplishments to the central server that signs them off, and today I'd like to talk about why it does that.

There are a bunch of little reasons why the Accomplishments system uses Ubuntu One, and one big reason. The big reason is this: running a service that serves lots of people is hard. Ask anyone in a startup; ask Twitter; ask anyone who has seen a webpage saying 504 Gateway Timeout or "We're experiencing a lot of traffic at the moment". If your service becomes popular, making it able to cope with the number of people using it is a difficult job. You need to have servers for the web service itself, servers to fail over onto if those get too busy or crash, servers to load-balance between them, servers to run haproxy, servers to be the front-end, servers to cope with unwrapping SSL, servers to run the underlying database, servers, servers, servers. It's hard work, and the state of the art evolves every single day. And this is because a web service is synchronous.

You may think here, "huh?". Take a step back. The Ubuntu Accomplishments system lets you get two* types of accomplishments: local and machine-verified. Local accomplishments are things that only occur on your machine: you've just beaten Solitaire, for example. We need not talk further of those here. Machine-verified accomplishments are those which you are awarded (by a script running on your machine which checks whether you've accomplished them) and then that award is verified by another machine, so you can't lie about having them; this is things like filing your first Launchpad bug, joining your Ubuntu loco team, and so on. The obvious way to build this system would be to have the script which runs on your machine check whether you've filed your first Launchpad bug, and then if you have, to request http://ubuntuac.com/verify?email=me@example.com&accom=first-launchpad-bug and have that server verify that you have indeed got it. However, imagine a world in which many many people have installed the accomplishments system; our poor ubuntuac.com server would be overwhelmed. We'd have to care about haproxy and load-balancing and front-ends and nagios and so on. Now, this might actually be OK for an Ubuntu service — the Canonical sysadmins are excellent at their jobs — but it ought to be possible to build things without needing a whole sysadmin team.

This is exactly the same issue that desktop programs have; if you try and do something which might take a long time, and then block waiting for the result, your application hangs. Reading a file will make your app go grey if it turns out to be a long file. The way to fix this is to be asynchronous, to request the file and have it load in the background and then call you back when it's done. Building a popular service is similar. The way to build such a service isn't to hit some verify URL and then wait for the result, because then the service has to rush to give you an answer, conscious that you're waiting for it. Be asynchronous. And that's what Ubuntu One brings to the Accomplishments system. Instead of verifying an accomplishment immediately, the Accomplishments system stores it in Ubuntu One, in a folder which is shared with the Accomplishments server. It'll then be synced to Ubuntu One, and then to the Accomplishments server; the server can verify it at its leisure and then sign the accomplishment, and that signature is then synced back to Ubuntu One and then down to your machine. None of these steps are harmed by waiting. If your computer goes offline halfway through the process, no problem: your accomplishment will be synced when you go back online. If the Accomplishments server has a million things to do, no problem; it works through them as fast as it can manage and each will be done in turn. If Ubuntu One itself crashes, no problem; your accomplishments will sync when it recovers. If the Accomplishments server dies, no problem; you're not waiting for a URL from it, so you'll not get a Gateway Timeout; instead, the server will be fixed from crashing and then just carry on with its list of files, and any accomplishments that were sent to it while it was dead will still arrive.

Of course, it would be possible to build a web service and an Accomplishments client which achieved this in other ways. Have the Accomplishments client detect whether you're online and queue up requests if you're not; have it detect if the connection drops while it's asking for verification; have the Accomplishments server receive a verify request URL and then return an "it's in my queue, here's your claim ticket" token and have the client periodically poll for that ticket or wait on a Comet connection. But that's still a lot of effort, and it's a lot of effort that Ubuntu One already does for you. The Accomplishments client can get all this network-y retries-y queue-y cleverness just by dropping a file in a folder, and doing it that way means that services can be built by normal mortals rather than super-network-experts. Go asynchronous and let someone else do the really difficult work for you. That's why the Accomplishments system uses Ubuntu One, and it's why it's able to exist and scale without having an army of sysadmins keeping it going, no matter how many people use it.

If you're interested in building services which work like this, chat to me, because I run the Ubuntu One app developer programme.