This weeks PhotoFriday theme is “Sky-High.” My first thought was a picture from a plane but then I thought it was likely to be a bit of a cliché. Bizarrely I didn’t see many other entrants with the same idea so I went with the above image, taken on my iPhone landing back in London.

Please also vote for my entry in last weeks challenge, “Summer Fun.” I’m entry number 279.

Related posts:

1. High Speed
2. Damaged
3. Blurred

Last year Yahoo! announced, with no notice, a significant change that had far reaching consequences for all third party applications including my iPhone program, Yummy. This is the third in a series of posts that discusses how I dealt with it.

We’ve already talked about most of the work, starting with an overview, the announcement, the low level technical challenges and the implementation (technical and UI). All that remains it to launch it, and that’s what we’re going to talk about today.

Availability

I switched my own Delicious account over to use my Yahoo! ID to shake out as many bugs as possible and I’d already had a few emails from people asking if they could help, so by the time I submitted the update to Apple it was in pretty good shape.

It launched on 13 December 2009, simultaneously for the full version and Yummy Browser. I decided against holding back for the free version.

Despite the volume of code that had changed, I decided to call is a “point” update (moving from version 2.3.1 to 2.3.2) because it didn’t really add any new user-level features. In hindsight maybe I should have made more of a fuss over it, since it was the first — and is still the only — Delicious iPhone app that supported Yahoo! IDs. But it felt wrong to kick my competitors over something that was just as outside their control as mine¹.

Lessons

So what have we learned through all this process?

I’ve learned how vulnerable Yummy is to outside influence. I had been planning a new feature release before the end of last year but this slipped until the first quarter of 2010 because of all the work that I have outlined above. I’m glad that I’ve kept “agile” (an overused word in the software industry), sticking to relatively small and frequent releases as this made it much easier to change from one feauture set (mostly what you see in the current Yummy 2.4 release) to another (OAuth).

But I think most of the lessons need to be taken on the other side, by Yahoo! From my point of view there were multiple failures:

1. No notice. It was impossible to have an application ready for day zero as we just didn’t know that a change was coming
2. No documentation. Even when we found out about the change there was little to no documentation about how to make the change
3. No concessions to desktop apps. It just is not possible to write a desktop application² with a fluid, easy to use UI. OAuth works nicely for web apps but not so well for desktop (and mobile) ones. Twitter is pushing forward with xAuth which solves this problem. It’s not yet standardised, which is a problem, but guaranteeing a poor user experience is hardly a winning strategy either
4. No transition. When Twitter moved from basic auth to OAuth³, both systems worked for over a year. This meant that there was no single, big switch over. True, they incentivised new apps to use the new scheme and the end of basic auth is going to orphan a bunch of apps but, still, there has been a year when developers can transition from one scheme to the other. For Delicious this change happened overnight, but not for all users. Instead we have to support both schemes and with no obvious method of knowing which one to use in advance

Luckily Yahoo! seem to have noticed the pain that their development community have been put through. They have since (albeit very recently) tried to get a handle on who is developing what against their APIs. Quite how they’ll use that information is anyones guess but it’s certainly a step in the right direction.

1. Six months later and with no updates from any of them I’m feeling less charitable!
2. By “desktop” I mean something that doesn’t run in a web browser. iPhone apps in this sense are “desktop” applications.
3. Past tense as, while this process is still ongoing, it will be complete shortly.

Related posts:

1. Delicious Debrief (Part 1/5)
2. Delicious Debrief (Part 2/5)
3. Delicious Debrief (Part 4/5)

Last year Yahoo! announced, with no notice, a significant change that had far reaching consequences for all third party applications including my iPhone program, Yummy. This is the third in a series of posts that discusses how I dealt with it.

On Monday I gave an overview of the problem, Tuesday I looked at how those changes were announced and why they were tricky, and yesterday I looked at how I actually implemented those technical details.

After the low level technical stuff there was still a lot of work making it useful. That’s what I’d like to talk about today.

But that’s not all

What I have been discussing so far has been entirely about the low-level, very technical details of OAuth. Of course there’s another side to it and that’s how it is presented to the user. This, again, has two parts: first, the options made available to the developer and, secondly, the work involved in actually making it happen.

So, the options available. We now have two authentication methods. I’ll call them Delicious and OAuth. The first point to make is that they’re not interchangeable. If I have an OAuth account I can’t log in with a Delicious username and password. And vice versa.

This begs the question: how can Yummy tell which type of account a user has?

The short answer is: it can’t.

Worse, many, if not most, Delicious.com users also have a Yahoo! account even when their delicious.com account is not linked to it. This results in a very confused user, as their login works without error yet they find they have no bookmarks.

What I was looking for was a way to identify the type of account that was accurate and would not require the same information to be entered multiple times. I ended up watching a number of people logging into their Yahoo! accounts¹ and, unfortunately, this invalidated most of my theories for making the process easier.

I thought, for example, that I was onto a winner when I noticed that I always typed “@yahoo.com” at the end of my username. Yummy could show the username screen as normal and, if the user typed “@yahoo.com” it would automatically remove the password field and offer to login using OAuth. However there are two problems with this. The minor, but annoying one, is that the user would be forced to enter their username twice, once in Yummy and then again in Safari. More significantly, I found that you don’t have to type “@yahoo.com” at the end of your Yahoo! ID. Although I do it all the time, most people seem to avoid the extra typing.

If people are going to avoid the extra typing, I would still need a way for them to override the default (Delicious) authentication method. And if they know that they have to overide it then, I reasoned, I may as well just come straight out and ask them how they’d like to proceed.

And this is exactly how it currently works.

Honestly, I’m really not very happy with it. How software talks to the server is entirely technical and is really not something that I should need to ask my users about. However this seems to be the least-bad approach. As I’ve noted before, I’m happy to entertain alternatives but I’ve not come across a better one yet.

The other significant user interface issue is one that I have already alluded to: should Yummy use a built-in web viewer or exit entirely and switch to Safari? The OAuth specification is clear on the matter: the users standard web browser should be used as it is both trusted and known². However, OAuth is really designed with web applications in mind. Even on a Mac this switch from “my” app to a web browser and back again would be far from ideal. But on an iPhone, that can only display a single application at a time, it is confusing at best.

All else being equal I would be happy to go against the advice of the specification and use an embedded web view if it made for a better user experience. Unfortunately, all else was not equal.

What I found was that, although I could log in correctly using the web view there was no way for Yummy to automatically regain control afterwards. If you’re familiar with OAuth you will be aware that there is supposed to be a method for doing this. I spent a long time trying to get this to work but never did. I’m unclear whether this was a problem in my code or on the Yahoo! side³.

In practice, this would have meant that I’d have to provide instructions before the user was allowed to log in. Something like “press the ‘Done’ button when you have finished logging in and enter the code you were given.” Of course this is not dramatically better than copying the code from Safari and relaunching Yummy.

In the end I decided that I would completely follow the spec rather than not follow its advice and force my users to jump through a few hoops.

Again, this is not something that I am entirely happy with. Could I have implemented this differently? Absolutely! Would the user experience be better had I done so? Not significantly.

In addition to the big things there were also a number of smaller, more cosmetic difficulties.

There a few things that that Delicious.com API is not terribly good at. One of the problems I’ve had from day one is that it’s not really designed for applcations to synchronise bookmarks (which is kind of bizarre when you think about it). It can be done, but it’s pretty tricky, especially when you have limited memory as is the case on the iPhone.

The new Yahoo! ID authentication scheme brought another: how do you know who has just logged in? Previously the user entered their username, so Yummy would know immediately. However in this case, their username is entered into Safari and Yummy never sees it. It so happens that one API call does return the username, but only under certain conditions and not necessarily straight away.

I mention this not because it was impossible to solve — indeed it was not spectacularly hard — but because it was was unexpected. After working through the low-level technical details of OAuth; and after figuring out how to allow the user to exit Yummy and return to the same point (while still allowing the process to be cancelled); after all that, the process was still not complete. What appeared to be just a different username and password became a significant and fundamental change.

Coming tomorrow

With most of the hard work finished I just had to make sure it really worked, and then sit back and analyse what went wrong.

1. Don’t worry, I wasn’t looking at what password you were typing.
2. We’re using an iPhone here so we can assume that the user is not using Internet Explorer.
3. I’ve got this working in the development version of Yummy now. I’m still not sure where the problem was.

Related posts:

1. Delicious Debrief (Part 1/5)
2. Delicious Debrief (Part 2/5)
3. Delicious Debrief (Part 5/5)

Last year Yahoo! announced, with no notice, a significant change that had far reaching consequences for all third party applications including my iPhone program, Yummy. This is the third in a series of posts that discusses how I dealt with it.

On Monday I gave an overview of the problem and yesterday I looked at how those changes were announced and why they were tricky. Today we’ll look at how I actually implemented those technical details — though not at the code level so don’t worry if you’re not a programmer!

Open Source OAuth

The two leading contenders are OAuthConsumer and MPOAuth. It’s only fair to note that my evaluation of both projects was in October and November 2009. Progress is likely to have been made on both projects since then.

I managed to get the sample projects for both up and running pretty quickly. However on trying to log in I immediately came across a problem. Yahoo! displayed a big warning message suggesting that you don’t authorise any new applications for security reasons. And, worse, that this method of authenticating was going to be removed in a few months.

Even if I could get it working this was going to be a show-stopper. I couldn’t in clear conscience ask users to ignore such a prominent message. It took a long time pouring over the code and the OAuth specification to realise that what changed was the OAuth specification itself.

As the error message noted, due to a security flaw in OAuth 1.0 they issued an update to the specification (called 1.0a) and it was this that Yahoo’s servers was objecting to. Neither library had been updated to the new version.

The change to both projects was small and straight-forward but I was worried that I actually needed to make any alterations at all. Were they being maintained? Was anyone using them?

It fairly quickly became clear that both projects were really optmised for connecting to, or at least developed against, Twitter. As with all standards, each implementation has their own peculiarities and specialisations. To be clear, there is nothing wrong with this. All code has its own axioms and its intended use defines its “shape.”

When I first looked at it, one of the things I liked about MPOAuth was that it did a lot of the hard work for you. It kept an internal state machine so that it knew which message it should be sending to the authentication server and what it should expect to see in return. I’m all for other people doing the work for me. As long as you followed the “correct” order everything worked just fine.

However, in a real world application you can’t expect users to do the right thing every time; there needs to be some way to recover from errors or allow people to change their minds. And, try as I might, I couldn’t quite figure out how to force MPOAuth to do what I needed. In trying to be clever it appeared to be making my life harder rather than easier.

OAuthConsumer got closer to being an integral part of Yummy. The way it’s structured was much more closely aligned with how I would have done it but I came across endless errors trying to connect to Yahoo’s servers. Most came back with very obscure errors and I spent a huge amount of time single-stepping (debugging) the framework — something that I’d hoped I could avoid by taking pre-existing code.

To be fair, much of this comes down to OAuth itself. On the server side there is no way to determine which parameter is incorrect, only that the request is not internally consistent.

OAuth in Yummy

Having spent a lot of time trying to understand why the existing libraries did not work with Yahoo! I had accumulated a good understanding of the OAuth 1.0a specification but a much less clear understanding of the frameworks themselves. There was something about the way that they had both been put together that didn’t quite mesh with either my way of thinking or the way that I was trying to implement it in Yummy — I’m still not quite sure which.

So I decided to go ahead and make my own OAuth library. In some respects this was quite a risky move as it was going to be difficult code to write and it would be in a very fundamental part of the software, traditionally somewhere that I try to use simple, well tested code.

It’s difficult to say whether this was the quickest option. Maybe I could have changed one line in, say, OAuthConsumer and got everything working in half the time. But the amount of time that I spent struggling with them suggests not and this way I have full control over and full understanding of every line of code¹.

I ended up writing about four hundred lines of new code instead of using MPOAuth or OAuthConsumer. This is some reasonably low-level code and was tricky to debug as the server-side typically responded with a simple “invalid signature” error no matter what the actual problem was. (This is not to complain about Yahoo’s service. It’s just the nature of the beast, unfortunately.) However, once written it’s proven to be surprisingly resilient².

Coming tomorrow

Having decided on the low-level technical details I discovered that there was still a lot of work to do. That’s what I’ll be talking about tomorrow.

1. I’m also geeky enough to like writing this kind of code. So shoot me.
2. I’m only aware of one bug caused by the new code, and that problem, strictly speaking, was already present in Yummy, it just did not exhibit the same behaviour previously.

Related posts:

1. Delicious Debrief (Part 2/5)
2. Delicious Debrief (Part 4/5)
3. Delicious Debrief (Part 5/5)

Last year Yahoo! announced, with no notice, a significant change that had far reaching consequences for all third party applications including my iPhone program, Yummy. This is the second in a series of posts that discusses how I dealt with it.

Yesterday I spoke at a high level about my iPhone application and some changes that Yahoo! made to their side of the system. Today I’d like to talk in a little more detail about how those changes were announced and why they were tricky.

The announcement

On 19 October 2009, Yahoo! posted a blog announcing that all new accounts would be accessed with a Yahoo! ID.

As far as I know, this is the first time that anyone outside Yahoo! knew about this change. And the change took effect immediately.

At the time of the announcement, Apple were taking around two weeks to approve updates to iPhone applications. This means that even if it took no time at all to make the required changes — which as we’ll see is far from the case — it would still mean that a user of Delicious.com would have approximately two weeks to wait before any third party application would support their new scheme.

I don’t necessarily expect Yahoo! to support me personally but I do expect them to act in ways that are beneficial to their own users. Knowing that no third party applications would work with new accounts clearly does not meet this goal.

Maintaining a healthy ecosystem around their own software with well maintained, documented and supported APIs seems to work to their advantage, yet they did not do this. Just look at the flourishing market for Twitter clients for an example of it done well.

But, put bluntly, they failed.

It took over a month to update the API documentation. And when they did they simply added a one paragraph note saying that authentication can now be performed using OAuth. Until that point the only official resources were a few responses in the Delicious discussion forums.

Now, months after the change went live, the documentation is still barely adequate to code against. I think the lack of other third party clients that support Yahoo! ID authentication reinforces this point. (At the time of writing, I think I’m right in saying that Yummy and Yummy Browser are the only two iPhone clients that support Yahoo! ID logins.)

Unfortunately the haphazard approach to developers was only the first part of the problem.

Surely it’s just a different username and password?

You’d think that using a Yahoo! password rather than a dedicated Delicious password would be pretty easy, but you’d be wrong.

OAuth, as it’s called, doesn’t work like that.

The idea is that you shouldn’t have to trust anyone other than Yahoo! with your Yahoo! password. So, bizarrely, I had to make Yummy work without ever knowing what your username or password is!

Without getting too bogged down in detail, this is roughly how it works:

1. The client launches a web browser. Since an iPhone can’t multi-task third-party applications, this means exiting and switching over to Safari¹. The changes announced for iPhone OS 4.0 don’t help here
2. The user logs into Yahoo! The server responds with a code
3. The user launches the client again and enters the code

Then every time the client wants to access a protected resource it does some cryptographic magic using the code (generating what’s called a “hash”) and sends that along with the request.

But the devil, as ever, is in the detail. The detail in this case is a twelve page document that assumes that you know what things like “octet string, first base64-encoded” means. Therefore in practice you end up looking up every second paragraph which slows things down.

Of course, writing the code yourself would be the hard option. For a standard such as OAuth there are bound to be some Open Source libraries that can be used without too much trouble.

Next…

Tomorrow I’ll talk about those Open Source libraries and what I ended up using.

1. The iPhone OS does allow applications to include an embedded web browser, indeed Yummy already has one to allow users to preview their bookmarks. However, the OAuth specification recommends against this approach for authentication. The idea is that you should only trust Yahoo! with your username and password and that entering them into a third-party application might break that trust. An early version of the change did include an embedded browser and I may yet revert back to that version.

Related posts:

1. Delicious Debrief (Part 4/5)
2. Delicious Debrief (Part 1/5)
3. Delicious Debrief (Part 5/5)

For the benefit of any new visitors, I develop an iPhone application called Yummy that connects to the Delicious.com “social bookmarking” website, allowing you to access and share your bookmarks using a far more usable interface than the native web or mobile optimised version of the site.

In the olden days — that is more than six months ago — to access Delicious all you had to do was enter a user name and password. As a developer it was very similar. Whenever you wanted to access something that required authorisation — adding a new bookmark for example — you had to supply the same credentials as you would on the web: a username and password. This, in web terminology, is called “basic authentication.”

And then in October last year, Yahoo! announced that any new accounts would be linked to their Yahoo! ID, the same username and password that are used to protect Yahoo! Mail, Flickr and most other Yahoo! services. A month later they allowed existing users to transition their old account to the new Yahoo! ID world.

On the Delicious.com website things look very similar. You still enter a username and password and can then access protected resources. But for developers things work a bit differently.

In the rest of this series of posts I’d like to talk about this transition and the difficulties involved. In the six months that it’s taken me to get around to writing this I’ve calmed down a lot but I’m sure you’ll be able to feel the frustration I felt at some points.

I’m going to split the discussion into a number of different “chapters.” First there’s how Yahoo! dealt with the transition. Next are the technical challenges, the options I tried and, eventually, the solution I settled on.

What follows is not densely technical, in the sense that I’m not going to be talking to the level of individual lines of code. However many of the decisions and trade-offs are somewhat technical in nature. Most people should be able to follow the process if not every last detail.

Tomorrow we’ll start with the announcement and why what they changed made things tricky for all third-party clients, including Yummy.

Related posts:

1. Delicious Debrief (Part 2/5)
2. Delicious Debrief (Part 4/5)
3. Delicious Debrief (Part 5/5)

Brighton Sunset

The phrase “Summer Fun,” which is this weeks PhotoFriday theme, conjures up images of children splashing in brightly coloured paddling pools, beach volley ball and barbecues lasting well into the late, twilight hours. Unfortunately I don’t have pictures of any of those things. So here is a picture of a summer sunset over Brighton beach. I think walking on the beach, late in the evening while it’s still light is considered fun.

Please vote for my entry in last weeks challenge, “Bloom.” I’m entry number 284.

Related posts:

1. Brighton Sunset
2. Seaside
3. The Beach

I was sat on the trading floor discussing a new feature that I was implementing with the person who would be using it the most.

“No, never.”

This was one detail of the change that would have far-reaching consequences in the code. A “no” would mean a few days of development, a “yes” would indicate several weeks.

“Are you absolutely sure? You don’t change it even once a month?”

I knew they’d like the smaller estimate but, equally, I knew that I didn’t want to end up trying to implement several weeks worth of functionality in the smaller time frame. I’ve seen that kind of thing happen too often.

“John¹, we don’t ever change these trades do we?”

John was the head trader. If anyone would know it would be him.

He rubs his head and leans back, thinking.

“Why would you want to? No, I can’t see us ever changing one.”

Of course, you can guess what happened thirty minutes into the first trading day with the new software.

So, what happened here? Why did we get the requirements so wrong? And why did we only find the mismatch after the system was moved into production?

Both the problem and the solution is the same thing: communication. Or more precisely, communicating using the right language.

Some developers are happy to only “speak” technical, proud that they are masters of their programming environment but ignorant of their users problems or how they really use the software.

Above I started out correctly, I was trying to understand the traders business and talking in terms of booking trades, positions, legal entities and a bunch of acronyms that would make even less sense out of context.

But I made one error: I used the word “change” without defining what I meant. I meant, well, any change. And so did they. Yet they didn’t consider moving a trade from one book to another to be a change and, unfortunately, I did.

You’d like to think that there were checks and balances in place to make sure that this kind of thing didn’t happen. And there were. In addition to informal and formal testing, there was over a week of “parallel running,” where the traders had to use the old and the new system together and check that the results were the same in both of them.

Were there any moved trades during this time? Of course. Why did the traders not notice? Well, it was about right; the differences, while present, were explicable and so not considered significant enough to mention even though I asked to hear about any problems at all.

So, again, communication. Or at least human nature. I wanted to hear about any differences but they tried to help me by only talking about differences that they couldn’t account for.

What’s the answer? Well, I’m not sure there’s an easy one. “Understanding your user” is a short, simple phrase but hides so much. If you spent the time to fully understood their job you probably wouldn’t have the time to do your own. But finding the balance is crucial.

1. Names have been changed to protect the… well, they work for a bank so I hesitate to say “innocent” but you know what I mean.

Related posts:

1. Random Changes
2. Blessed is the Tool Maker
3. Naked

When I ask, what I usually find is that these people have never actually used it. They have probably looked at screen shots, maybe one of the screencasts. They can see that there are few graphics and little animation.

That’s fair — it’s absolutely true — but that’s not what I think of when I think of polish.

Instead, Yummy is designed to be quick and convenient to use. It’s modelled more after Apple’s Contacts app than, say, the Camera or Photos. In this sense, it’s not a few well placed graphics that makes all the difference. It’s the fact that Yummy saves your location in it when you exit¹, even when you’re in the middle of editing a bookmark. It’s that your searches are automatically saved so you don’t need to re-type terms you use regularly. It’s that you can edit bookmarks even when offline. It’s that if you use a short bookmark (like tinyurl or bit.ly), you can fetch the full URL and the title of the web page.

In short it’s lots of little things that I can’t even begin to list on the features screen in its iTunes store listing. They’re not really features as such, but their absence would make for a much weaker program.

None of these things can be seen just by looking at screenshots. Indeed, some of those refinements may not be noticed at all. But I think it’s this very subtlety that defines polish. Marco Arment touches on this when he says that the best UI is invisible — this is exactly what I was aiming for.

1. Of course iOS4 does a lot of this for you now, but I first implemented this when iPhone OS 2.x was current.

Related posts:

1. Yummy: Prepare for Launch
2. Growing Up in Public
3. Yummy: Ready for Sale

Another late entry for PhotoFriday (this is becoming a theme in and of itself). This weeks image is around the theme “Bloom” which I felt pretty much had to be about flowers. These were in Bath, UK.

Please also vote for my entry in last weeks challenge, “Motionless.” I’m entry number 167.

Related posts:

1. Active
2. The Team
3. Sky-High