Dare Obasanjo aka Carnage4Life

Developers on Google Wave

2009-06-04T08:11:59.6596581-07:00

I initially planned to write up some detailed thoughts on the Google Wave video and the Google Wave Federation protocol. However the combination of the fact that literally millions of people have watched the video [according to YouTube] and I’ve had enough private conversations with others that have influenced my thinking that I’d rather not post something that makes it seem like I’m taking credit for the ideas of others. That said, I thought it would still be useful to share some of the most insightful commentary I’ve seen on Google Wave from various developer blogs.

Sam Ruby writes in his post Google Wave

At one level, Google Wave is clearly a bold statement that “this is the type of application that every browser should be able to run natively without needing to resort to a plugin”. At to give Google credit, they have been working relentlessly towards that vision, addressing everything from garbage collection issues, to enabling drag and drop of photos, to providing compelling content (e.g., Google Maps, GMail, and now Google Wave).
…
But stepping back a bit, the entire and much hyped HTML5 interface is just a facade. That’s not a criticism, in fact that’s generally the way the web works. What makes Google Wave particularly interesting is that there is an API which operates directly on the repository. Furthermore, you can host your own server, and such servers federate using XMPP.

These servers are not merely passive, they can actively interact with processes called “robots” using HTTP (More specifically, JSON-RPC over POST). Once invoked, these robots have access to a full range of operations (Java, Python). The Python library implementation looks relatively straightforward, and would be relatively easy to port to, say Ruby.

This dichotomy pointed out by Sam is very interesting. One the one hand, there is the Google Wave web application which pushes the boundaries of what it means to be a rich web application that simply uses Javascript and the HTML DOM. This is a companion step in Google’s transition to taking an active role in the future of building Web applications where previous steps have included Google representatives drafting the HTML 5 specification, Google Gears and Google Chrome. However where things get interesting is that the API makes it possible to build alternate client applications (e.g. a .NET Wave client written in C#) and even build services that interact with users regardless of which Wave client they are using.

Joe Gregorio has more on these APIs in his blog post Wave Protocol Thoughts where he writes

There are actually 3 protocols and 2 APIs that are used in Wave:

Federation (XMPP)

The robot protocol (JSONRPC)

The gadget API (OpenSocial)

The wave embed API (Javascript)

The client-server protocol (As defined by GWT)

The last one in that list is really nothing that needs to be, or will probably ever be documented, it is generated by GWT and when you build your own Wave client you will need to define how it talks to your Wave server. The rest of the protocols and APIs are based on existing technologies.

The robot protocol looks very easy to use, here is the code for an admittedly simple robot. Now some people have commented that Wave reminds them of Lotus Notes, and I'm sure with a little thought you could extend that to Exchange and Groove. The difference is that the extension model with Wave is events over HTTP, which makes it language agnostic, a feature you get when you define things in terms of protocols. That is, as long as you can stand up an HTTP server and parse JSON, you can create robots for Wave, which is a huge leap forward compared to the extension models for Notes, Exchange and Groove, which are all "object" based extension models. In the "object" based extension model the application exposes "objects" that are bound to locally that you manipulate to control the application, which means that your language choices are limited to those that have bindings into that object model.

As someone’s whose first paying job in the software industry was an internship where I had to write Outlook automation scripts to trigger special behaviors when people sent or modified Outlook task requests, I can appreciate the novelty of moving away from a programming model based on building a plugin in an application’s object model and instead building a Web service and having the web application notify you when it is time to act which is the way the Wave robot protocol works. Now that I’ve been exposed to this idea, it seems doubly weird that Google also shipped Google Apps Script within weeks of this announcement.

Nick Gall writes in his post My 2¢ on Google Wave: WWW is a Unidirectional Web of Published Documents -- Wave is a bidirectional Web of Instant Messages that

Whether or not the Wave client succeeds, Wave is undoubtedly going to have a major impact on how application designers approach web applications. The analogy would be that even if Google Maps had "failed" to become the dominant map site/service, it still had major impact on web app design.

I suspect this as well. Specifically I have doubts about the viability of the communications paradigm shift that Google Wave is trying to force taking hold. On the other hand, I’m sure there are thousands of Web developers out there right now asking themselves "would my app be better if users could see each other’s edits in real time?","should we add a playback feature to our service as well" [ed note - wikipedia could really use this] and "why don’t we support seamless drag and drop in our application?". All inspired by their exposure to Google Wave.

Finally, I've ruminated publicly that I see a number of parallels between Google Wave and the announcement of Live Mesh. The one interesting parallel worth calling out is that both products/visions/platforms are most powerful when there is a world of different providers each exposing their data types to one or more of these rich user applications (i.e. a Mesh client or Wave client). Thus far I think Google has done a better job than we did with Live Mesh in being very upfront about this realization and evangelizing to developers that they participate as providers. Of course, the proof will be in the pudding in a year or so when we see just how many services have what it takes to implement a truly interoperable federated provider model for Google Wave.

Now Playing: Eminem - Underground/Ken Kaniff

A closer look at the upcoming RSS Bandit ribbon

2009-05-31T13:32:06.8898383-07:00

In between watching the Google Wave video and Slumdog Millionaire, I got around to completing the first set of tabs for the ribbon in RSS Bandit. Screenshots are below, as usual let me know what you think.

Fig 1: The home tab. This is the default tab on launching the application. I like that formerly hidden features of the application like subscribing to newsgroups and managing podcasts are now front and center without having to compromise on the common tasks that people want to perform.

Fig 2: The ability to synchronize RSS Bandit with your Google Reader or NewsGator Online feeds is also now a lot more discoverable instead of being hidden in some obscure menu with an obscure name ("Synchronize Feeds").

Fig 3: The folder tab. This is menu is contextual and becomes selected when you click on a folder in the tree view. There are two features I’d like to call out in this view; Rules and Filters.

Fig 4: The rules tool is where we’ll end up placing existing and new options on behavior the user would like executed on receipt or viewing of new content.

Fig 5: The filter tool is used for filtering the items that show up in the list view. We've had several requests for this feature over the past few years but couldn’t figure out an elegant way to incorporate it into the user interface.

Fig 6: The feed tab. This is a contextual tab that is selected when you click on a feed in the tree view. One feature that I love which is now properly highlighted is that we support creating new posts in feeds that support this such as newsgroups (existing feature) or posting a new status update on Facebook if you have hooked it up as a feed source (new feature).

Fig 7: The item tab. This is the contextual tab that is highlighted when you select an item in the list view. There are no new features highlighted here. What we do think will be interesting is if we make it straightforward for existing and new IBlogExtension plugins to end up showing up in the item tab. So you should think of this tab as being extensible and should expect that some of our existing plugins (e.g. "Email This", "Post to Twitter", etc) will also end up in this tab.

Twitter: We’re all celebrities now

2009-05-27T08:36:09.6759642-07:00

A few days ago, Jeff Atwood responded to one of my status messages on Twitter with the following response of his own

This is a good opportunity to talk about what Twitter brings to the table as a social software application (as opposed to the Twitter as Google Killer meme). Twitter currently positions itself as a microblogging platform which is implies that it’s like blogging just smaller. A blog is often two things, first of all it’s about personal publishing platform for one or more people to share their opinions and knowledge with the world. The second thing is that it is the community of people who read that blog and the conversations they have about it on the site. The second is usually embodied by comments on the blog. In fact some, like Jeff Atwood, have argued that a blog without comments isn’t really a blog. As Jeff writes

I firmly maintain that a blog without comments enabled is not a blog. It's more like a church pulpit. You preach the word, and the audience passively receives your evangelical message. Straight from God's lips to their ears. When the sermon is over, the audience shuffles out of the church, inspired for another week. And there's definitely no question and answer period afterward.

Of course, I'm exaggerating for comedic effect. Maybe a blog with comments disabled is more analogous to a newspaper editorial. But even with a newspaper editorial, readers can make public comments by sending a letter to the editor, which may be published in a later edition of the paper.

When you look at a blog such as Mashable and compare it its Twitter counterpart or even Jeff Atwood’s blog versus his Twitter account, it seems clear which is more of church pulpit where the audience passively receives your evangelical message versus a forum for two way communication between the audience and the author.

An interesting dynamic that Twitter has added to personal publishing that doesn’t have a good analog in blogging is the notion of a public list of subscribers to the publisher’s content with links to every one of them and a fairly pejorative name for them “followers”. This feature has led to both micro and macro celebrities engaging in games to see who can amass the most fans with the most notable public display being the race between Ashton Kutcher and CNN to a million followers.

Twitter takes blogging to the next level as a platform for building and encouraging celebrity. The other side of this is poignantly captured in James Governor’s post A truth of Asymmetric Follow: On sadness, fans and fantasy

Well last week I had a chance to walk in the fan’s shoes, and of course I learned a lot, while trying to build buzz for our charitable efforts for Red Nose Day. I have to admit I hated it. I *really* wanted to get the attention of @wossy or @stephenfry. Could I? Of course not. These guys have day jobs…

But it was only on spending a lot of time surfing around user profiles to check for spambots that I discovered how profoundly depressing the celebrities on Twitter phenomenon can be. It was coming across profiles of Twitter users following ten or so celebrities on Twitter (and nobody else), wondering why their questions weren’t being answered. Why are they ignoring me, I keep asking them questions? After I saw a few of these profiles I felt a little depressed.

From this perspective it is unsurprising that tech celebrities like Jeff Atwood & Robert Scoble and real-world celebrities like Ashton Kutcher & John Mayer love the Twitter dynamic. Similarly, it is also unsurprising that over 60% of users abandon the service within the first month. After all, we aren’t all celebrities.

In its current form, Twitter is growing primarily as a platform for celebrities, wannabe celebrities and their fans. The key thing to note is that celebrity here isn’t limited to the kind of people you read about in People magazine and US Weekly. For example, I use Twitter to follow web technology celebrities like Tim O'Reilly and Scott Hanselman. On the other hand, my wife uses Twitter follows popular mommy bloggers like McMommy and Playground for Parents.

Going back to Jeff Atwood’s twitter message, I don’t consider Twitter to be Facebook with the annoying bullshit stripped out. For the most part, the Facebook experience has focused on being away to bring your offline relationships to the web. This is captured in the current home page design which proclaims that Facebook helps you connect and share with the people in your life.

From my perspective, this goal has more widespread appeal and utility than being a next generation platform for celebrity on the Web. Your mileage may vary.

Now Playing: Kid Cudi - Day N Nite (remix) (feat. Jim Jones & Trey Songz)

Preview of Facebook Comments in RSS Bandit and the Ribbon

2009-05-24T22:15:54.5241611-07:00

I’ve made some more progress in integrating the Facebook news feed into the next version of RSS Bandit currently codenamed Colossus. This weekend I completed the addition of support for viewing and replying to comments in the news feed. So here are some screenshots of current comment workflow for interacting with Facebook comments

Fig 1: Viewing the comments in response to a funny status update from Anil Dash

Fig 2: Responding to the comment by pressing "Ctrl + R" or right-clicking and selecting Post Reply.

Fig 3: The news feed on Facebook with the comment posted from RSS Bandit

The second major change coming in the Colossus release is the adoption of the design elements from the Microsoft Office fluent user interface such as the ribbon, contextual tabs, galleries and live preview. To prepare for this change, we’re first building a prototype of the redesigned user interface and once we’re happy with it we will start refactoring the RSS Bandit application to enable swapping out our existing menus and taskbars with the new interface.

Here’s where we are in the design prototype for next release. Let me know what you think in the comments.

OpenID’s Tipping Point

2009-05-22T06:54:53.0157356-07:00

In the past week or so, two of the biggest perception problems preventing proliferation of OpenID as the de facto standard for decentralized identity on the Web have been addressed. The first perception problem is around the issue of usability. I remember attending the Social Graph Foo Camp last year and chatting with a Yahoo! employee about why they hadn’t become an Open ID relying party (i.e. enable people to login to Yahoo! account with OpenIDs). The response was that they had concerns about the usability of OpenID causing reducing the number of successful log-ins given that it takes the user off the Yahoo! sign-in page to an often confusing and poorly designed page created by a third party.

Last year’s launch and eventually success of Facebook Connect showed developers that it is possible to build a delegated identity workflow that isn’t as intimidating and counterproductive as the experience typically associated with delegated identity systems like OpenID. On May 14th, Google announced that a similar experience has now been successfully designed and implemented for OpenID in the Google Code blog post titled Google OpenID API - taking the next steps which states

We are happy to announce today two new enhancements to our API - introducing a new popup style UI for our user facing approval page, and extending our Attribute Exchange support to include first and last name, country and preferred language.
The new popup style UI, which implements the
OpenID User Interface Extension Specification, is designed to streamline the federated login experience for users. Specifically, it's designed to ensure that the context of the Relying Party website is always available and visible, even in the extreme case where a confused user closes the Google approval window. JanRain, a provider of OpenID solutions, is an early adopter of the new API, and already offers it as part of their RPX product. As demonstrated by UserVoice using JanRain's RPX, the initial step on the sign-in page of the Relying Party website is identical to that of the "full page" version, and does not require any changes in the Relying Party UI. >

Once the user selects to sign in using his or her Google Account, the Google approval page is displayed. However, it does not replace the Relying Party's page in the main browser window. Instead it is displayed as a popup window on top of it. We have updated our Open Source project to include a complete Relying Party example, providing code for both the back-end (in Java) and front-end (javascript) components.

Once the user approves the request, the popup page closes, and the user is signed in to the Relying Party website.

The aforementioned OpenID User Interface Extension allows the relying party to request that the OpenID provider authenticate the user via a “pop up” instead of through navigating to their page and then redirecting the user back to the relying party’s site. Thus claim that OpenID usability harms the login experience is now effectively addressed and I expect to see more OpenID providers and relying parties adopt this new popup style experience as part of the authentication process.

The second biggest perception blocker is the one asked in articles like Is OpenID Being Exploited By The Big Internet Companies? which points out that no large web companies actually support OpenID as a way to login to their primary services. The implication being that companies are interested in using OpenID as a way to spread their reach across the web including becoming identity providers for other companies but don’t want others to do the same to them.

That was true until earlier this week when Luke Shepard announced Facebook Supports OpenID for Automatic Login. Specifically,

Now, users can register for Facebook using their Gmail accounts. This is a quicker, more streamlined way for new users to register for the site, find their friends, and start exploring.

Existing and new users can now link their Facebook accounts with their Gmail accounts or with accounts from those OpenID providers that support automatic login. Once a user links his or her account with a Gmail address or an OpenID URL, logs in to that account, then goes to Facebook, that user will already be logged in to Facebook.

In tests we've run, we've noticed that first-time users who register on the site with OpenID are more likely to become active Facebook users. They get up and running after registering even faster than before, find their friends easily, and quickly engage on the site.

This makes Facebook the first major web company to truly embrace OpenID as a way to enable users to sign up and login to the site using credentials from a third party (a competitor even). The fact that they also state that contrary to popular perception this actually improves the level of engagement of those users is also a big deal.

Given both of these events, I expect that we’ll see a number of more prominent sites adopting OpenID as they now clearly have nothing to lose and a lot to gain by doing so. This will turn out to be a great thing for users of the web and will bring us closer to the nirvana that is true interoperability across the social networking and social media sites on the web.

Wolfram Alpha: The Wikipedia Killer

2009-05-16T09:05:31.6555994-07:00

After all the hype, I got around to taking Wolfram Alpha for a spin last night due to being unable to sleep after a weird Doctor Manhattan themed nightmare. The experience of using the site is very impressive and there is a great walkthrough of the power of the site in the Wolfram Alpha screencast which I encourage people to watch if you are interested in learning about a new breed of search engine.

There have been a ton of articles calling Wolfram Alpha a "Google Killer" but after using the site for a few hours although I find it fascinating, I question how much of a threat the site is to Google either as a way to satisfy the typical questions people ask Web search engines or a threat to Google’s search advertising cash cow. You can get a sense for the kinds of queries that Wolfram Alpha handles amazingly well from the list below

As you can tell from the above list, Wolfram Alpha is like having a search engine over the kind of data you’d see in the CIA's World Factbook or Time Almanac. There really isn’t anything like it on the Web today. However it isn’t really a competitor to traditional web search engines who for the most part are still focused on finding web pages despite the various advancements in answering a subset of queries with direct answers instead of links to web pages such as Google's OneBox results and Live Search’s instant answers feature.

From my perspective, the threat to search engines like Google isn’t Wolfram Alpha but the trend it represents. That trend is the renaissance of the vertical search engine. Earlier this year, I was putting together a panel at the MIX ‘09 conference and needed to invite the panelists from a pool of people who I’d either heard about or knew of professionally but had never contacted directly. How did I find out how to contact these people? Even though all of them had blogs, there wasn’t a consistent way to track down contact information. So I looked them up on Facebook and sent each of them a private message. Mission accomplished. Unbeknownst to me, Facebook had become my “people” search engine”.

Here’s another story. Last year I worked on the most satisfying software release of my career, Windows Live (wave 3). After the launch I wanted to find out what people were saying about the product so I did a Twitter search for Windows Live and posted the results. While I wasn’t paying attention, Twitter had become my “what are people saying about <insert brand here>” search engine.

This trend of search engines dedicated to specific scenarios and contexts that can’t be answered well by Web search is the trend that traditional search engines should watch carefully.

I can imagine Wolfram Alpha eventually growing to satisfy a lot of the sorts of queries I go to Wikipedia today to get answers to and doing so in a more authoritative manner. In that case, it would become my “facts and trivia” search engine. However there are currently too many gaps in its knowledge of commercial products (e.g. search for “ipod” results in a coming soon notice) and people (e.g. the Jim Carrey entry is amazingly brief yet still manages to have a factually inaccuracy) to make it a true replacement for wikipedia. That said, the service shows great promise and it will be interesting watching as it evolves.

Why Twitter's Engineers Hate the @replies feature

2009-05-15T06:51:42.523742-07:00

Biz Stone, Twitter’s , recently wrote in a blog post entitled The Replies Kerfuffle that

We removed a setting that 3% of all accounts had ever touched but for those folks it was beloved.
…
97% of all accounts were not affected at all by this change—the default setting is that you only see replies by people you follow to people you follow. For the 3% who wanted to see replies to people they don't follow, we cannot turn this setting back on in its original form for technical reasons and we won't rebuild it exactly the same for product design reasons.
…
Even though only 3% of all Twitter accounts ever changed this setting away from the default, it was causing a strain and impacting other parts of the system. Every time someone wrote a reply Twitter had to check and see what each of their followers' reply setting was and then manifest that tweet accordingly in their timeline—this was the most expensive work the database was doing and it was causing other features to degrade which lead to SMS delays, inconsistencies in following, fluctuations in direct message counts, and more. Ideally, we would redesign and rebuild this feature but there was no time, hence the sudden deploy.

As someone whose day job is working on a system for distributing a user’s updates and activities to their social network in real-time across Web and desktop applications, I’m always interested in reading about the implementation choices of others who have built similar systems. However when it comes to Twitter I tend to be more confused than enlightened whenever something is revealed about their architecture.

So let’s look at what we know. When Ashton Kutcher posts an update on Twitter such as

it has to be delivered to all 1.75 million of his followers. On the other hand, when Ashton Kutcher posts an update directed to one of his celebrity friends such as

then Twitter needs to decide how to deliver it based on the Replies settings of users.

One option would to check each of the 1.75 million followers of aplusk’s setting to decide whether they need have @replies restricted to only people they are following. Since this will be true for 97% of his followers (i.e. 1.7 million people) then there would need to be a 1.7 million checks to see if the intended recipient are also friends of John Mayer before delivering the message to each of them. On the other hand, it would be pretty straightforward to deliver the message to the 3% of users who want to see all replies. Now this seems to be what Biz Stone is describing as how Twitter works but in that case the default setting should be more expensive than the feature that is only used by a minority of their user base.

In that case I’d expect Twitter to argue that the feature they want to remove for engineering reasons is filtering out some of the tweets you see based on whether you are a follower of the person the message is directed to not the other way around.

What have I missed here?

Update: A comment on Hacker News put me on track to what I probably missed in analyzing this problem. In the above example, if the default case was the only case they had to support then all they have to do to determine who should receive Ashton Kutcher’s reply to John Meyer is perform an intersection of both user’s follower lists. Given that both lists need to be in memory for the system to be anywhere near responsive, performing the intersection isn’t as expensive as it sounds.

However with the fact that 3% of users will want to have received the update even though they aren’t John Mayer’s friends means Twitter needs to do a second pass over whoever was not found in both follower lists and check what their @reply delivery settings are. In the above example, even if every follower of John Mayer was a follower of Ashton Kutcher, it would still require 750,000 settings checks. Given that it sounds like they keep this setting in their database instead of in some sort of cache, it is no surprise that this is a feature they’d like to eliminate.

Are you building an application that people use every day or do you just think you are?

2009-05-11T08:02:57.1921027-07:00

Joshua Porter has an excellent post titled Are you building an everyday app? (the LinkedIn problem) where he writes

In a recent interview, LinkedIn CEO Reid Hoffman describes moving away from day to day to a more strategic role in the company he founded:

I want to be able to sink my mind around a couple of problems and work through them. For example, many professionals still don’t understand how LinkedIn can be valuable on a daily or weekly basis”

Another way you could phrase this is: “people don’t use LinkedIn everyday…we need to figure out how to change that”.

The fact is that LinkedIn, in its current incarnation, is not an everyday app. An everyday app is one that is used every day (or most days) by its users.
...
In general, most people think they’re building an everyday app, but they’re not. When the actual use patterns are discovered, most apps will be used every few days or less. Designers have to ask themselves a very hard question: “How often are people really going to use our web application?”. The answer is important…it will even help drive design decisions. Whether or not you have an everyday app affects the entire design of what you’re building, including the screens, notifications, and frequency of the service. For example, only everyday apps really need to use real-time technology to update streams. If you find out that you’re not building an everyday app, you probably don’t need to invest in making it real-time. But…you might invest in a notifications system that can alert users to when something very interesting happens.

You don’t have to be an everyday app to be successful. Netflix, for example, is not an everyday app. It’s an every-few-days app. Most people go back every few days to update their queue. There is really no need to go back more often.

Many developers of social software applications on the Web believe they’ve built an everyday app but they actually haven’t. One thing I’ve learned in almost five years of working on social software applications at Microsoft is that simply having the features of an everyday app doesn’t translate to people using the application every day. The best way to think about this is that no application starts off as an everyday app. Very rarely does an application show up that is so amazing that people start using it everyday right off the bat. Instead there is a transition where either users transition from occasional to frequent users while the application stays static or the application itself transitions to catering to a more engaged user base relative to where it was in previous years.

An example of the former is Twitter, the site hasn’t really changed much since I started using it about a year and a half ago. However it wasn’t until the right set of factors came together such as getting enough people I was following, adding the Twitter app to Facebook & Windows Live to update my friends on those services and getting Twitter applications for my desktop & phone did I transition to becoming an everyday user. Twitter’s main problem is that not every user eventually hits this sweet spot which is why you read articles like Twitter Quitters Post Roadblock to Long-Term Growth which points out that retention rate over a one month period hovers between 30% – 40% for new users. This need to make the application instantly useful to users is what prompts features like the Suggested Users List whose purpose is to give new users content worth coming back to every day instead of the “Trying out Twitter for the first time” style posts that they probably see from most of their friends who are also kicking the tires on the service they heard about on Oprah.

Having features that are useful every day like a constantly updating activity stream doesn’t mean people will use the site every day. For the users that cross the hump, it does. The challenge is how to get users to cross that hump.

A site that has done a good job of motivating its users to check it out on a daily basis and adjusting its features as its user base has become more engaged is Facebook. One of Facebook’s most annoying features for a long time was the fact that notifications on new messages in the service didn't actually contain the message. I suspect the purpose of doing this was to drive users back to the site so that they would then catch up on all they’d missed such as invitations and content in the news feed after they were done answering the message. Although this feature is annoying, it was definitively effective given anecdotal feedback from various people I talked to at the time. After a certain point, Facebook’s user engagement grew to the point where sending messages without the content to drive users back to the site wasn’t worth it relative to the decreased customer satisfaction.

Another great example from Facebook is the transition from the old news feed to the new stream. In March of 2009, the news feed was transformed into a real-time stream and almost two months later the real-time stream now updates live without having to refresh the page. The previous functionality of the news feed was relegated to an alternate highlights stream as shown below

From reading the blog posts about the changes, the problem the switch from a highlights-centric news feed to a real-time stream is addressing is the fact that the highlights-based feed is stale and doesn’t provide enough value for users who’ve not just become every day users but are now every hour users. Not only do over 100 million of its users login every day but with 90 million users generating 90 billion page views in the month of March 2009 that implies the average page view for a Facebook user is over once per hour.

And when you think about it, the introduction of the original news feed in 2006 was a successful attempt to go from being an occasionally updated rolodex for a large chunk of their users into a social utility to keep up with what’s going on in the lives of family, friends and coworkers. The switch to a real-time stream is how Facebook is addressing the fact that they have slowly become an every hour app instead of just an everyday app for their users.

A number of services I use online could learn from how Facebook has evolved their experience over time increase the engagement of their user base by making sometimes small and sometimes huge changes to the user experience to encourage people to make the service a regular part of their lives.

Some thoughts on using the Facebook Open Stream API

2009-05-11T08:02:33.6300551-07:00

In my previous post I talk about adding support for reading and commenting on your Facebook news feed in RSS Bandit. This functionality is made possible by the recently announced Facebook Open Stream API. As I worked through the code in my few free moments I was alternately impressed and frustrated by the Open Stream API. On the one hand, the functionality the API provides is greatly enabling and has already unleashed a bounty of innovation as evidenced by the growing number of applications for interacting with your Facebook news feed on the desktop (e.g. Seesmic, Tweetdeck, bdule, etc). On the other hand, figuring out there are a few quirks of the API that make the web developer in me cringe and the desktop developer want to beg for mercy.

Below are my opinions on the Open Stream API, the purpose of sharing is so other developers who plan to use the API may avoid some of the pitfalls I did and also to share feedback with my peers at Facebook and elsewhere on best practices in providing activity stream APIs.

Good: Lots of options for getting at the data

Most API calls at Facebook have two entry points. You can either call a “REST-like” URL endpoint via HTTP GET or perform a SQL-like query via FQL on a generic end point. For interacting with the Facebook news feed, you have stream.get and stream.getComments as the REST-like methods for accessing the news feed and a comment thread for a particular feed item respectively. With FQL, you can perform queries against the stream (FQL) and comment (FQL) to get the same results.

Both mechanisms give you the option of getting back the data as XML or JSON. Below are example of what HTTP requests to retrieve your news feed look like using both approaches

stream.get REQUEST:
http://api.facebook.com/restserver.php?v=1.0&method=stream.get&format=XML&viewer_id={0}&session_key={1}&api_key={2}&call_id={3}&sig={4}

FQL REQUEST:
http://api.facebook.com/restserver.php?v=1.0&method=fql.query&format=XML&call_id={0}&session_key={1}&api_key={2}&sig={3}&query={4}
FQL QUERY:
select post_id, source_id, created_time, actor_id, target_id, app_id, message, attachment, comments, likes, permalink, attribution, type from stream where filter_key in (select filter_key FROM stream_filter where uid = {0} and type = 'newsfeed') and created_time >= {1} order by created_time desc limit 50

I ended up using stream.get over FQL because the only benefit I see to using FQL is being able to filter some of the result fields or combine data from multiple tables, neither of which I needed in this instance. I chose XML as the output format over JSON because I transform the results of the Facebook API request into an Atom feed and then hand it down to the regular Atom feed parsing code in RSS Bandit. It was much easier for me to handle this transformation using XSLT over XML results as opposed to procedural code over a JSON result set.

There’s also a third mechanism for getting news feed data out of Facebook that would have been perfect for my needs. You can access the news feed using the nascent activity streams standard which returns the data as an Atom feeds with certain extensions specific to social network activity stream. I couldn’t get this to work probably due to user error on my part (more on this later) but if I had, the structure of the GET request would have been along the lines of

Activity Streams REQUEST:
http://www.facebook.com/activitystreams/feed.php?source_id={0}&app_id={1}&session_key={2}&sig={3}&v=0.7&read&updated_time={4}

If not for the initial problems I had figuring out which parameters to pass to APIs and the concern about building on an API that isn’t yet at version 1.0, the Activity Stream API would have been perfect for my needs.

Kudos to the Facebook team for providing such a rich and varied set of options for getting the news feed out of their system. There’s something for every temperament.

Bad: Misuse of HTTP status codes

The Facebook API documentation describes the platform as “REST-like” and not RESTful. They sure weren’t kidding. For the most part, when I’ve discussed the problems with APIs that aren’t completely RESTful the negatives to such an approach have seemed aesthetic in nature as opposed to being practical problems. Below are some of the practical problems I faced because the Facebook APIs do not use HTTP status codes in a manner consistent with the rest of the Web.

In HTTP, there are existing error codes which clients can interpret in a consistent manner and provide consistent feedback to users when they occur. Now consider this excerpt from the documentation on using Facebook’s activity stream API

Response Codes

Like most HTTP responses, Facebook Activity Streams responses include a response header, which always includes a traditional response code and a short response message. The supported response codes include:

200 Code provided whenever the Facebook servers were able to accommodate the request and provide a response.

304 Code provided whenever the request header included If-Modified-Since and no new posts have been generated since the specified time.
Note: Code 304 will never be returned if If-Modified-Since isn't included in the request header.

401 Code provided whenever the URL omits one or more of the required parameters.

403 Code provided whenever the URL is syntactically valid, but the user hasn't granted the required extended permission.

404 Code provided whenever the URL is syntactically valid, but the signature is incorrect, or the session key is invalid.

You might notice that an HTTP 401 is used to indicate that the request is improperly formed. However, let’s see what RFC 2616 has to say about the 401 status code as well as how to communicate badly formed arguments

400 Bad Request

The request could not be understood by the server due to malformed syntax. The client SHOULD NOT repeat the request without modifications.

401 Unauthorized

The request requires user authentication. The response MUST include a WWW-Authenticate header field (section 14.47) containing a challenge applicable to the requested resource. The client MAY repeat the request with a suitable Authorization header field (section 14.8). If the request already included Authorization credentials, then the 401 response indicates that authorization has been refused for those credentials. If the 401 response contains the same challenge as the prior response, and the user agent has already attempted authentication at least once, then the user SHOULD be presented the entity that was given in the response, since that entity might include relevant diagnostic information. HTTP access authentication is explained in "HTTP Authentication: Basic and Digest Access Authentication"

According to the HTTP specification excerpted above, the status code to return on missing parameters should be 400 not 401. The practical problem with returning HTTP 401 in this case is that applications like RSS Bandit may have code paths that prompt the user to check or re-enter their credentials because an authentication error has occurred. We now have to special case getting a 401 from Facebook’s servers versus any other server on the Web. Thankfully, this error should be limited to development time unless Facebook changes their API in a backwards incompatible manner by requiring new parameters to their methods.

A bigger problem is that Facebook returns HTTP 200 which traditionally means success in regular occurring error conditions. Specifically, when a user’s session key expires a successful response is sent containing an error document. In RSS Bandit, we already have a processing pipeline that hands off successful responses containing an XML document to the RSS/Atom parsing layer. With Facebook’s behavior I had two choices

Modify the RSS/Atom parsing layer to understand Facebook error documents and then kick back an error up to the user interface asking the user to re-enter their credentials. This was particularly hacky because that layer doesn’t really have a connection to the main UI thread nor should it.
Pre-process the XML document before handing it to the RSS/Atom parsing layer. This implies an intermediate step in between dispatching on the response status and actually processing the document.

Option #2 proved more palatable since there was already an intermediate step needed to transform the results of stream.get to an Atom feed. If I’d used the Activity Streams API as was my initial plan then the decision may have been harder to make.

Good: Well thought out model for authorizing access to user data

A number of application platforms work by giving a user all-or-nothing access to the user’s data. My favorite example of this bad practice is Twitter which for a long time made this situation even worse by requiring the applications to collect the person’s username and password. The controversy around Twply earlier this year shows exactly why giving applications more access to a user’s data than they need is bad. Twply only needed access to a user’s @replies but given that there was no way to only give it access to just that aspect of a user’s Twitter data, it also got the ability to post tweets on their behalf and did so in a spammy manner.

The Facebook API has a notion of extended permissions which are access rights that require special opt-in from the user given that the data or functionality is sensitive and can be abused. The current set of extended permissions in the Facebook API is provided below

Permission	Description
publish_stream	Lets your application or site post content, comments, and likes to a user's profile and in the streams of the user's friends without prompting the user. This permission is a superset of the status_update, photo_upload, video_upload, create_note, and share_item extended permissions, so if you haven't prompted users for those permissions yet, you need only prompt them for publish_stream. Note: At this time, while the Open Stream API is in beta, the only Facebook users that can grant your application the `publish_stream` permission are the developers of your application.
read_stream	Lets your application or site access a user's stream and display it. This includes all of the posts in a user's stream. You need an active session with the user to get this data.
email	This permission allows an application to send email to its user. This permission can be obtained only through the fb:prompt-permission tag or the promptpermission attribute. When the user accepts, you can send him/her an email via notifications.sendEmail or directly to the proxied_email FQL field.
offline_access	This permission grants an application access to user data when the user is offline or doesn't have an active session. This permission can be obtained only through the fb:prompt-permission tag or the promptpermission attribute.
create_event	This permission allows an app to create and modify events for a user via the events.create, events.edit and events.cancel methods.
rsvp_event	This permission allows an app to RSVP to an event on behalf of a user via the events.rsvp method.
sms	This permission allows a mobile application to send messages to the user and respond to messages from the user via text message.
status_update	This permission grants your application the ability to update a user's or Facebook Page's status with the status.set or users.setStatus method. Note: You should prompt users for the publish_stream permission instead, since it includes the ability to update a user's status.
photo_upload	This permission relaxes requirements on the photos.upload and photos.addTag methods. If the user grants this permission, photos uploaded by the application will bypass the pending state and the user will not have to manually approve the photos each time. Note: You should prompt users for the publish_stream permission instead, since it includes the ability to upload a photo.
video_upload	This permission allows an application to provide the mechanism for a user to upload videos to their profile. Note: You should prompt users for the publish_stream permission instead, since it includes the ability to upload a video.
create_note	This permission allows an application to provide the mechanism for a user to write, edit, and delete notes on their profile. Note: You should prompt users for the publish_stream permission instead, since it includes the ability to let a user write notes.
share_item	This permission allows an application to provide the mechanism for a user to post links to their profile. Note: You should prompt users for the publish_stream permission instead, since it includes the ability to let a user share links.

Bad: Too many prompts for desktop applications

Although I’ve praised the extended permissions model, it currently leads to a cumbersome experience for desktop applications. Installing Facebook desktop applications like bdule or Facebook for Adobe Air requires running through three separate permissions screens. The user has to login, then grant the read_stream extended permission followed by the publish_stream extended permission. Granted, the latter two only need to be done once but they still affect the out of box experience fairly negatively in my opinion.

In RSS Bandit, I’ve attempted to reduce this by delaying the prompt for publish_stream permission until the first time a user tries to comment on a news feed item from within the application. Streamlining this experience will be a boon for application developers who want the entire experience to be smooth and painless. The documentation on the documentation the Open Streams API page states that there are options for streamlining these requests but they only apply to Web applications not desktop apps.

Bad: Plethora of application identifiers and authentication requirements is a stumbling block for beginners

The official documentation on the Facebook API doesn’t do a good job of connecting all the dots when it comes to understanding how to make calls to the service. For example, when you read the documentation for stream.get it is not obvious from that page what endpoint to make requests to OR that every Facebook API call has a set of required parameters beyond the ones listed on that page. In fact, I was stumped for about a week or so trying to figure out the magical incantations to get the right set of parameters for API calls and the right data to put in these parameters until I stumbled on a Facebook forum post creating the 'sig' parameter which solved my problems. I believe I once saw this information in the Facebook API documents but after ten minutes of searching just now I can’t seem to find it so perhaps it was in my imagination.

Part of the problem is the varied number of identifiers that you have to keep straight as an application developer including your

application ID
API key
application secret
session key
client secret

The fact that various APIs take different combinations of the above lead to more than one confusing moment for me. Eventually I figured it out but I felt like I was being hazed as I was going through the process.

Progress report on Facebook integration into RSS Bandit (Colossus release)

2009-05-10T09:43:35.0796426-07:00

So I’ve spent the weekend working my way through Facebook’s Open Stream API and have made a ton of progress in adding the option to read your Facebook news feed from RSS Bandit. In my previous post I should the initial screenshots where you can login to Facebook via RSS Bandit to begin the authentication process to retrieve your news feed. Below are a few more screenshots showing how much progress has been made

Fig 1: This is the screen where you give RSS Bandit permission to access your news feed. Note that this is different from the screen where you login to Facebook via RSS Bandit. You should only see this screen once but will get the login screen quite often unless you select the option to stay signed into Facebook via RSS Bandit.

Fig 2: This is what your Facebook news feed looks like in RSS Bandit with the default stylesheet for Facebook applied. As you can see from the screen shot I’ve attempted to replicate the look of the news feed as best as I can in RSS Bandit. This includes options below each item to comment or see who has liked an item.

Fig 3: There is also support for viewing the comments on a Facebook item inline within the Feed Details list view. Although this is different from how Facebook does it, I felt this was needed to make it consistent with the other inline comment viewing experiences within RSS Bandit.

There’s still a bunch of cleanup to do such as fix up the look of comments when you click on them and provide a way to post comments on your friends’ Facebook items. This is probably another day or so of work which I’ll do next weekend. After that it is back to fixing bugs and working with Torsten on our idea from over two years ago on how to add a Office 2007 style ribbon to the application.

Below are pictures of the initial prototypes which Torsten has dusted off and will start working on again

I’ll follow up this post with some of my thoughts on writing a desktop application that targets Facebook’s Open Stream API. The process was definitely more painful than I expected but the results are worth it which says a lot about the functionality provided by the API.

RSS readers modeled after email clients are fundamentally broken

2009-05-05T05:17:10.4391887-07:00

Farhad Manjoo has an article on Slate entitled Kill your RSS reader which captures a growing sentiment I’ve had for a while and ranted about during a recent panel at SXSW. Below are a few key excerpts from Farhad’s article that resonate strongly with me

In theory, the RSS reader is a great idea. Many years ago, as blogs became an ever-larger part of my news diet, I got addicted to Bloglines, one of the first popular RSS programs. I used to read a dozen different news sites every day, going to each site every so often to check whether something fresh had been posted. With Bloglines, I just had to list the sites I loved and it would do the visiting for me. This was fantastic—instead of scouring the Web for interesting stories, everything came to me!
...
But RSS started to bring me down. You know that sinking feeling you get when you open your e-mail and discover hundreds of messages you need to respond to—that realization that e-mail has become another merciless chore in your day? That's how I began to feel about my reader. RSS readers encourage you to oversubscribe to news. Every time you encounter an interesting new blog post, you've got an incentive to sign up to all the posts from that blog—after all, you don't want to miss anything. Eventually you find yourself subscribed to hundreds of blogs, many of which, you later notice, are completely useless. It's like having an inbox stuffed with e-mail from overactive listservs you no longer care to read.

It's true that many RSS readers have great tools by which to organize your feeds, and folks more capable than I am have probably hit on ways to categorize their blogs in a way that makes it easy to get through them. But that was just my problem—I began to resent that I had to think about
organizing my reader.

This mirrors my experience of that of many of my friends who used to be enthusiastic users of RSS readers. Today I primarily find out what’s going on in blogs using a combination of Twitter, Techmeme and Planet Intertwingly. The interesting thing is that I’m already subscribed to about half of the blogs that end up getting linked to in these sources on a regular basis yet I tend to avoid firing up my RSS reader.

The problem is that the RSS readers I use regularly, Google Reader and RSS Bandit, take their inspiration from email clients which is the wrong model for consuming casual content like blogs. Whenever I fire up an email application like Outlook or Hotmail it presents me with a list of tasks I must complete in the form of messages that need responses, work items, meeting invitations, spam that needs to deleting, notifications related to commercial/financial transactions that I need to be aware of and so on. Reading email is a chore where you are constantly taunted by the BOLD unread messages indicator silently nagging you about the stuff you haven’t done yet.

Given that a significant percentage of the time, the stuff in my email inbox is messages that were sent directly to me that need some form of response or acknowledgment this model is somewhat sound although as many have pointed out there is a lot of room for improvement.

When it comes to blogs and other casual content, this model breaks down. I really don’t need a constant nagging reminder that I haven’t read the half dozen reposts of the same tech news stories about Google, Twitter and Facebook after I’ve seen the first one. Furthermore, if I haven’t fired up my reader in a while then I don’t care to be nagged about all the stuff I missed since they are just blogs so it is OK if I never read them. This opinion isn’t new, Dave Winer has been evangelizing “River of News” style aggregators for several years and given the success of this model for social networking sites like Facebook and microblogging sites like Twitter, it’s clear that Dave was onto something.

Looking back at the time I’ve spent working on RSS Bandit, I realize there are a couple of features I added to attempt to glom the river of news model on top of an email based model for reading feeds. These features include

the ability to mark all items as read after navigating away from a feed. This allows you to skim the interesting headlines then not have to deal with the “guilt” of not reading the rest of the items in the feed.
a reading pane inspired by Google Reader where unread items are presented in a single flow and marked as read as you scroll past each item

Looking back now, it seems to me that the way we think of RSS readers needs to fundamentally change. Presenting information as a news feed where the user isn’t pressured to read every item or feel like a failure is one way to move the needle on the user experience here. What I wonder is whether it isn’t already too late for this category of applications as services like Twitter & Facebook take over as how people keep up to date with what’s going on with the people and content they care about.