Dare Obasanjo aka Carnage4Life

Joe Hewitt on Irony

2009-11-14T07:03:28.1184728-08:00

Joe Hewitt, the developer of the Facebook iPhone application, has an insightful blog post on the current trend of developers favoring native applications over Web applications on mobile platforms with centrally controlled app stores in his post On Middle Men. He writes

The Internet has been incredibly empowering to creators, and just as destructive to middle men. In the 20th century, every musician needed a record label to get his or her music heard. Every author needed a publishing house to be read. Every journalist needed a newspaper. Anyone who wanted to send a message needed the post office. In the Internet age, the tail no longer wags the dog, and those middle men have become a luxury, not a necessity.

Meanwhile, the software industry is moving in the opposite direction. With the web and desktop operating systems, the only thing in between software developers and users is a mesh of cables and protocols. In the new world of mobile apps, a layer of bureacrats stand in the middle, forcing each developer to queue up for a series of patdowns and metal detectors and strip searches before they can reach their customers.
...
We're at a critical juncture in the evolution of software. The web is still here and it is still strong. Anyone can still put any information or applications on a web server without asking for permission, and anyone in the world can still access it just by typing a URL. I don't think I appreciated how important that is until recently. Nobody designs new systems like that anymore, or at least few of them succeed. What an incredible stroke of luck the web was, and what a shame it would be to let that freedom slip away.

Am I the only one who thinks the above excerpt would be similarly apt if you replaced the phrase "mobile apps" with "Facebook apps" or "OpenSocial apps"?

Now Playing: Lady GaGa - Bad Romance

Does OAuth Have a Dark Side?

2009-11-07T08:41:08.6693177-08:00

There was an article in the The Register earlier this week titled Twitter fanatic glimpses dark side of OAuth which contains the following excerpt

A mobile enthusiast and professional internet strategist got a glimpse of OAuth's dark side recently when he received an urgent advisory from Twitter. The dispatch, generated when Terence Eden tried to log in, said his Twitter account may have been compromised and advised he change his password. After making sure the alert was legitimate, he complied.

That should have been the end of it, but it wasn't. It turns out Eden used OAuth to seamlessly pass content between third-party websites and Twitter, and even after he had changed his Twitter password, OAuth continued to allow those websites access to his account.
…
Eden alternately describes this as a "gaping security hole" and a "usability issue which has strong security implications." Whatever the case, the responsibility seems to lie with Twitter.

If the service is concerned enough to advise a user to change his password, you'd think it would take the added trouble of suggesting he also reset his OAuth credentials, as Google, which on Wednesday said it was opening its own services to work with OAuth, notes here.

I don't think the situation is as cut and dried as the article makes it seem. Someone trying to hack your account by guessing your password and thus triggering a warning that your account is being hacked is completely different from an application you've given permission to access your data doing the wrong thing with it.

Think about it. Below is a list of the applications I've allowed to access my Twitter stream. Is it really the desired user experience that when I change my password on Twitter that all of them break and require that I re-authorize each application?

I suspect Terence Eden is being influenced by the fact that Twitter hasn't always had a delegated authorization model and the way to give applications access to your Twitter stream was to handout your user name & password. That's why just a few months ago it was commonplace to see blog posts like Why you should change your Twitter password NOW! which advocate changing your Twitter password as a way to prevent your password from being stolen from Twitter apps with weak security. There have also been a spate of phishing style Twitter applications such as TwitViewer and TwitterCut which masquerade as legitimate applications but are really password stealers in disguise. Again, the recommendation has been to change your password if you fell victim to such scams.

In a world where you use delegated authorization techniques like OAuth, changing your password isn't necessary to keep out apps like TwitViewer & TwitterCut since you can simply revoke their permission to access your account. Similarly if someone steals my password and I choose a new one, it doesn't mean that I now need to lose Twitter access from Brizzly or the new MSN home page until I reauthorize these apps. Those issues are ~~orthogonal~~ unrelated given the OAuth authorized apps didn't have my password in the first place.

Thus I question the recommendation at the end of the article that it is a good idea to even ask users about de-authorizing applications as part of password reset since it is misleading (i.e. it gives the impression the hack attempt was from one of your authorized apps instead of someone trying to guess your password) and just causes a hassle for the user who now has to reauthorize all the applications at a later date anyway.

Now Playing: Rascal Flatts - What Hurts The Most

Startup Advice: Find Underserved Markets

2009-11-04T07:09:35.2398827-08:00

“Every marketer's dream is to find an unidentified or unknown market and develop it” – Barry Brand

Last week I read the various notes on the presentations by famous startup founders at Startup School 2009 and found a lot of the anecdotes interesting but wondered if they were truly useful to startup founders. I tried to think of some of the products I started using in the past five years that I now use a lot today and couldn’t do without. In looking for the common thread in these products and the quote above came to mind.

Finding an underserved market sounds like getting the winning ticket in a lottery, unlikely. In truth it isn’t hard to find underserved markets if you recognize the patterns. The hard part in turning it into a successful business is execution.

There are two patterns I’ve used in looking for underserved markets that were invigorated by products I now use regularly today. The first pattern is looking for activities people like doing where the technology has been stagnant for a while. For the past few decades, we’ve been in world where the technology products you use can be made smaller, cheaper and faster every few years. There is some notion of waiting for the stars to align such as hard drive sizes shrinking until you could fit gigabytes of data in your pocket (e.g. the iPod) or AJAX and online banking becoming ubiquitous (e.g. Mint). However for each product category where some upstart has changed the game by leveraging modern technologies there are still dozens of markets where decade(s) old tech is dictating the user experience.

Another patterns is looking for an activity or task that people hate doing but assume is a fact of life or a necessary aspect of using a product. I remember when I’d buy video games like Soul Calibur and lament about how I could only find people to play with when I went to visit friends from out of town who were also fans of the game. To me, not being able to play multiplayer games without having friends physically in the same room was just a fact of live. This is no longer the case in the world of XBox Live. I also remember almost cancelling my cable subscription about five or six years ago because I resented having to tailor my TV watching schedule around prime time hours which to me represented prime hours to decompress from work and hack on RSS Bandit. The entire notion of “prime time TV” has been a fact of life for decades. Once I discovered TiVo, it stopped being the case for me. Some of these painful facts of life have been with us for centuries. For example, take this quote from John Wanamaker

“Half the money I spend on advertising is wasted; the trouble is I don't know which half”

with modern advertising solutions like Google’s Adwords and Facebook’s Advertising platform this is no longer the case. People can now tell down to minute levels of detail exactly what their return on investment is on advertising.

I’d love to see more startups attempting to find and satisfy underserved markets instead of going with the crowd and doing what everyone else is doing. Less Facebook games and iPhone fart apps, more original ideas that solve real problems like Mint and Flickr. If your startup needs a few hints at what some of these markets are in the technology space, there’s always YCombinator’s list of Startup Ideas they’d fund.

Now Playing: Jay-Z Feat. Kanye West - Hate

New MSN Homepage with Activity Streams from Windows Live, Facebook and Twitter

2009-11-04T06:10:55.6315264-08:00

On the MSN blog there's a new blog post entitled The New MSN Homepage Unveiled which states

Today is an exciting day for our team at MSN because we unveiled the most significant redesign our MSN.com homepage has seen in over a decade. We spent thousands of hours talking with customers; testing hundreds of ideas; experimenting around the world and carefully evaluating what our users want, and don’t want - to deliver a homepage that is designed to be the best homepage on the Web. We hope you’ll agree.
…
So, we started from scratch to cut the clutter on our homepage and reduced the amount of links by 50%. There’s also a simplified navigation across news, entertainment, sports, money, and lifestyle that lets you drill into information topics that interest you, without being overwhelming. Local information from your neighborhood is important to you and so is high quality, in-line video – so we offer both, right on the homepage. And, you told us you want the latest information not only from your favorite sources, but also from your friends, and the breadth of the Web – so we now offer convenient access to Facebook, Twitter, & Windows Live services and the most powerful search experience on the Web from Bing, empowering you to make more informed, faster decisions. And this is just the beginning - keep visiting our blog for more MSN news in the coming weeks.

This is a really exciting release for my team on Windows Live since we're responsible for the underlying platform that powers the display of what activities your friends have been performing across Windows Live. Working with the MSN home page team was a good experience and its great to see that the tens of millions of people who visit the MSN home page regularly will now get to experience our work. Kudos to the MSN team on a very nice release.

You can try out the new home page for yourself at http://preview.msn.com

Now Playing: Jay-Z - Reminder

Real-time, Distributed Conversations: Some Thoughts on the Salmon Protocol

2009-11-02T06:50:44.5589111-08:00

Last week John Panzer, who works on Blogger at Google, wrote about some of the work he’s been doing on creating a protocol for syndicating comments associated with activity streams in his post The Salmon Protocol: Introducing the Salmon Project. Key parts of his post are excerpted below

A few days ago, at the Real Time Web Summit, we had a session about Salmon, a protocol for re-aggregated distributed conversations around web content. I was hoping for some feedback and to generate some interest, and I was overwhelmed by the positive reactions, especially after Louis Gray's post "Proposed Salmon Protocol aims to unify Conversations on the Web". Adina Levin's "Salmon - Re-assembling distributed conversations" is a good, insightful review as well. There's clearly a great deal of interest in this, and so I've gone ahead and expanded Salmon's home at salmon-protocol.org with an open source project, salmon-protocol.googlecode.com, and a mailing list, groups.google.com/group/salmon-protocol.

Louis Gray’s post on the topic includes an embedded presentation which captures the essence of the protocol

Before talking about the technical details of the protocol it is a good idea to understand the end user problem the protocol solves. For me, it solves a problem I have in the way that RSS Bandit integrates with Facebook. The problem is that although there is a way to get regular updates on changes to the user’s news feed by polling Facebook’s stream and getting data back in the Activity Stream format there isn’t a mechanism today to get updates on the comments on items in the feed. What it means in practice today is that once an item rolls off of the news feed, there is no way to keep the comments up to date in RSS Bandit.

The Salmon Protocol aims to address this problem by piggybacking on PubSubHubBub as a way for applications to get real-time updates on comments on items in an activity stream not just updates on new activities.

There have also been several mentions of Salmon being a way to aggregate distributed conversations on an item (e.g. this blog post is syndicated to FriendFeed and there are comments there as well as in the comments on my blog) but I am less clear on those scenarios or whether Salmon is enough to solve the various tough problems that need to be solved to make that work end to end.

Any API for posting comments to a site needs to solve two problems; identity and dealing with comment spam. I decided to take a look at the Salmon Protocol Summary to see how it addresses these problems.

The meat of the Salmon Protocol format is excerpted below

A source provides an RSS/Atom feed of content. It includes a Salmon link in its feed:

<link rel="salmon" href="http://example.org/salmon-endpoint"/>

An aggregator reads the feed (ideally via a push mechanism such as PubSubHubbub), and sees from the link that it is Salmon-enabled. It remembers the endpoint URL for later use.

When an aggregator's user leaves a comment on a feed item, the aggregator stores the comment as usual, and then also POSTs a salmon version of it to the source's Salmon endpoint:

POST /salmon-endpoint HTTP/1.1

Host: example.org

Content-Type: application/atom+xml

<?xml version='1.0' encoding='UTF-8'?>

    <entry xmlns='http://www.w3.org/2005/Atom'>

    <author>

      <name>John Doe</name>

      <uri>acct:johndoe@aggregator-example.com</uri>

    </author>

    <content>Yes, but what about the llamas?</content>

    <id>tag:aggregator-example.com,2009:cmt-441071406174557701</id>

    <updated>2009-09-28T18:30:02Z</updated>

    <thr:in-reply-to xmlns:thr='http://purl.org/syndication/thread/1.0'

       ref='tag:example.org,1999:id-22717401685551851865'/>

    <sal:signature xmlns:sal='http://salmonprotocol.org/ns/1.0'>

        e55bee08b4c643bc8aedf122f606f804269b7bc7

    </sal:signature>

    <title/>

</entry>

The commenter is identified in the published comment using the atom:uri element. How this author is authenticated in situations outside of public comments on a blog such as RSS Bandit posting a comment to Facebook on my behalf isn’t really discussed. I noticed an offhand reference to OAuth headers which seems to imply that the publishing application should also be sending authentication headers as well when publishing the comment. How these authentication headers would flow through the systems involved is unclear to me especially given the approach Salmon has taken to deal with spam prevention.

The workflow for dealing with spam comments is described as follows

A major concern with this type of distributed protocol is how to prevent spam and abuse. Salmon provides building blocks to allow in-depth defense against attacks. Specifically, every salmon has a verifiable author and user agent. The basic security flow when salmon swims upstream looks like this:

aggregator-example.com: "Here is a salmon, authored and signed by 'acct:johndoe@aggregator-example.com'; please accept it."

Recipient: "I know that this is really aggregator-example.com due to its OAuth headers, and it has a good reputatation, but I do not trust it completely; I will do a double check."

Recipient: Uses Webfinger/XRD to discover salmon validation service for acct:johndoe@aggregator-example.com, which turns out to be hosted by aggregator-example.com.

Recipient: "Given that johndoe has delegated Salmon validation to aggregator-example, and I know I'm talking to aggregator-example already, I'll skip the actual check." (Returns HTTP 200 to aggregator-example.com)

The flow can get more complicated, especially if the aggregator is not also providing identity services for the user. In the most general case, the recipient needs to take the salmon, discover a salmon validator service for the author via XRD discovery on the author's URI, and POST the salmon to the validator service. The validator service does an integrity / signature check against the salmon and returns 200 if the salmon checks out, 400 if not. The signature check means that the given author (johndoe in this case) signed the salmon with the given id, parent id, and timestamp. It does not attempt to do a full, XML-DSig style verification, though such a service is another reasonable extension.

This flow seems weird and it is unclear to me that it actually solves the problems involved in distributed commenting. So let’s say I post a comment to Facebook from RSS Bandit, in step 3 above they are now supposed to use WebFinger to lookup my email address provider and determine which service I use for digitally signing comments. Then they ask it if the comment looks like it was from me.

Hmm, this looks like a user authentication workflow in disguise as a comment validation workflow. Shouldn’t the service receiving the comment (i.e. Facebook) in the example above be responsible for validating my identity not some third party service? Maybe this protocol wasn’t meant for sites like Facebook?

Let’s say this protocol is really meant for situations when the comment recipient doesn’t intend to be the sole identity provider such as commenting on Robert Scoble's blog where he allows comments from anyone with just an email address and an optional web page URL as identifiers. So each commenter needs to provide an email address on an email service provider that supports WebFinger and validates digital signatures in the specific situation related to the Salmon protocol? Sounds like boiling the ocean. I wonder why this can’t work with OpenID validation or some other authentication protocol that has already been validated by developers and is seeing some adoption?

At the end of the day, I think the problem Salmon attempts to solve is one that needs solving as activity streams become a more popular and intrinsic feature across the Web. However in its current form it’s hard for me to see how it actually solves the real problems that exist today in a practical way.

Of course, this may just be my misunderstanding of the protocol documents currently published and I look forward to being corrected by one of the protocol gurus if that is the case.

Now Playing: Chris Brown - I Can Transform Ya (feat. Lil Wayne)

Keeping the Stream Pure: FriendFeed vs. Twitter vs. Facebook

2009-11-02T06:00:42.9570785-08:00

Robert Scoble had an epiphany on one of the key problems with FriendFeed’s design that he now realizes now that he’s no longer a fanatical user of the service in his post The chat room/forum problem (& an apology to @Technosailor) . Robert writes

Twitter got lists.

This let us throw together a list of experts. For instance, I put together a list of people who have started companies. Compare that feed to your average Facebook feed and you’ll see it in stark black and white: your Facebook feed is “fun” but isn’t teaching you much.

It becomes even more stark when you do a list like my tech news brands list. See, this is NOT a forum! It is NOT a chat room!

No one can enter this community without being invited. Now compare to FriendFeed. We could have built a list like this over there, but it would have gotten noiser because of a feature called “Friend of a Friend.” That drags in people the list owner didn’t invite. Also, anyone can comment underneath any items on Facebook or FriendFeed. That brings people into YOUR life that YOU DID NOT INVITE!

Again, at first, this seems very democratic and very nice. After all, it’s great to throw a party for the whole world and let them drink your wine and have conversations with your kids. But, be honest here, would you rather have a private dinner with Steve Jobs, or would you rather have a dinner with Steve Jobs and 5,000 people who you don’t really know?

As someone who works on a platform for real-time streams for a living I always find it interesting to compare the approaches of various companies.

I agree with Robert that FriendFeed’s friend of friend feature breaks a fundamental model of the stream. In an earlier post on the problems with some of FriendFeed's design decisions I pointed out the following problem with the feature

FriendFeed shows you content from friends of friends: This is major social faux pas. It may sound like a cool viral feature but showing me content from people I haven't subscribed to means I don't have control of who shows up in my feed and it takes away from the intimacy of the site because I'm always seeing content from strangers.

One of the things I’ve learned about how people interact with activity streams and news feeds is that it is important to feel like you are in control of the experience. FriendFeed’s friend of friend feature explicitly takes that away from users. I can understand why they did it (i.e. to increase the amount of content showing up in the stream for people with few friends and as a friend discovery mechanism) but it doesn’t change the fact that the behavior can seem like a nuisance and even lead to lamebook style socially awkward situations.

However unlike Robert I don’t really agree with his characterization of the differences between streams on Facebook versus Twitter. On both sites, I as a user choose who the primary sources that show up in my stream are. Facebook is for people I know, Twitter is for brands and people I find interesting. I find both sites fun but I agree that I’m more likely to learn something new related to work from Twitter than from Facebook. Whether I am “learning” something new or not isn’t what’s important but whether I feel like I’m getting value out of the experience. As Biz Stone wrote in a blog post on Twitter's new terms of service

At the start, critics often said, "Twitter is fun, but it's not useful." At one point @ev responded dryly with, "Neither is ice cream."

Although comments in the news feed on Facebook bring people I didn’t explicitly add into my stream, they are often OK because they are often people who I consider to be part of my extended social network or at least are on topic. This would never work on Twitter with retweets being shown inline (for example) since anyone can follow anyone else or retweet their content which would lead to the same sort of chat room/forum noise that Robert decries in his post.

Looking at the sketches from Twitter’s Project Retweet

I don’t think Robert’s concerns about retweets polluting the stream are warranted. From the above sketch, it just looks like Twitter is fixing the bug in the retweeting process where I have to use part of my 140 character quota to provide attribution when retweeting an interesting status update. This leads to interesting behavior such as people keeping their tweets under 125 characters to enable retweeting. I have to admit I’ve wondered more than once if I should make a tweet shorter so that it is easier to retweet. Project Retweet is fixing in a way that encourages an existing user practice on the site. That deserves kudos in my book.

Now Playing: Jay-Z Feat. Alicia Keys - Empire State Of Mind

Facebook Seattle Engineering Road Show: Mike Shroepfer on Engineering at Scale at Facebook

2009-10-29T08:17:12.1494067-07:00

Earlier today (or should I say yesterday) I attended Facebook’s Seattle Engineering Road Show which was a part technical talk and part recruiting event where Mike Shroepfer and a number of other Facebook engineers gave a fairly deep technical talk about the technologies used by Facebook.

Below are my notes on the presentation and ensuing Q&A session. One thing I found interesting about the talk was how similar their architecture for the platform that powers the Facebook news feed is to what my team has built for powering the Windows Live What's New feed.

Before the presentation started there were a bunch of slides that carried some interesting and sometimes impressive stats about Facebook including

Average number of friends per user is 130
Total number of Facebook applications is 350,000
It is the number 1 Photo Site on the Web
More events are created on the site than on Evite per day
People spend 6 8 billion minutes per day on Facebook which makes it the number 1 site on the Web according to many measurement services including Nielsen and ComScore.
The fastest growing demographic is users aged 35 and older.
The page with the most fans is Barack Obama with 6.8 million fans
The TV show with the most fans is South Park.
70% of their users are outside the United States of America

Mike Shreopfer started his talk with a number of impressive statistics. These stats include the fact that 2 billion pieces of content are shared per week when you add up all of the status updates, comments, likes and other sharing gestures that are performed on the site. There are also 2 billion photos uploaded per month, 15,000 Facebook Connect implementations which in combination with the various platform applications make over 5 billion API calls per day and about 300 million active users per month.

Scaling Facebook has proven more challenging than scaling a traditional website. In a traditional website, you put a user’s data in a database then whenever the user’s data is requested you have some web or mid-tier layer retrieve the user’s data from the database. Such applications (e.g. Web-based email services) are easily partitionable and it is thus straightforward to scale them horizontally by creating different pools of users and putting various groups of users on particular servers. The original design for Facebook was similarly trivial to scale. In the early days, each college lived on it’s own cluster (e.g. http://harvard.facebook.com) and the service was thus trivial to scale. This model broke down when Facebook made it possible for users to make friends from different colleges and eventually making friends across different geographical networks. At this point it became harder to partition data since all users were interconnected. Rendering a user’s homepage didn’t just mean looking up that user’s data but also data from all their friends. This change continues to be Facebook’s core scalability challenge.

The most famous example of tackling this challenge is the Facebook news feed. When you hit the news feed they render 45 stories on your homepage. The service has a couple of milliseconds to query the feeds of up to 5,000 of your closest friends and then choose which 45 stories are the best ones to show the user. There is a platform named Multifeed which is the underlying platform which powers the news feed and allows them to filter down tens of thousands of stories to forty five stories when rendering the home page. This process involves fetching metadata for news stories from memcache clusters that perform over 50 million operations per second. These queries include rich media such as photos which are served at the rate of 1.2 million photos per second.

In aggregate they currently store more than 20 billion photos at 4 different resolutions per photo (i.e. 80 billion image files). Back in 2005, photos were served from NFS clusters. They eventually hit some bottlenecks because file systems are not really designed for serving large numbers of photos. The metadata ended up being too large to fit in memory and there were numerous I/O bottlenecks. The second implementation of the photos service contained several implementations such as putting profile photos in a caching tier and moving a large number of photos to a Content Distribution Network (CDN). They also built a file handle cache for their NFS systems to reduce the amount of disk I/O involved when looking up the location of the file. The 3rd generation of their photo service is a custom storage solution known as Haystack. Haystack runs on commodity hardware (i.e. Dell Linux boxes with a terabyte SATA drives) and is a custom file system which keeps all file indexes in RAM to speed up lookups. Today Haystack is so optimal that it only takes one I/O operation to load a photo compared to 3 I/O operations for the previous optimized implementation and 10 I/O operations for the initial NFS based system. Haystack is so scalable that they once accidentally disabled their CDNs and Haystack served all photos on the site without breaking a sweat. The service was built by 3 developers in a couple of months. It will be Open Sourced in the near future.

Another example of Facebook’s approach to scaling is the recent addition of custom usernames to the site. When the feature launced they had over 200 million active users. There was a lot of internal debate on how to fairly allocate usernames to their users and they even tested auctioning usernames. However this ended up being confusing to users so they decided to go with the simpler first come, first served model as the primary way for users to get their preferred user name. Given the expected rush of users who would sign up for username, the developers expected the load to be the equivalent of a denial of service attack and planned accordingly. There were a number of contingency plans to reduce load on the site including disabling the chat feature, reducing the number of items in the news feed from 45 to 15 and even serving static HTML on some pages. However none of these contingencies had to be put in place when the username feature was launched because the service scaled so well. Users signed up for 200,000 usernames in the first 3 minutes, 500,000 within the first 15 minutes and had created over 1 million usernames within the first hour. Since usernames were created at the root of the Facebook namespace, many users created “fun” usernames such as http://www.facebook.com/home.aspx and they actually had manually reject a user who registered a common typo to one of their pages.

At the high level you can think of their architecture as being as follows.

There are PHP web servers which then aggregate data from dedicated web services, in-memory caches and their relational database. They use PHP on the front-end because it is easy to learn, easy to develop in and it is very productive. However there are downsides such as the fact that their benchmarks have shown that it is ten times slower than if they used more traditional web programming technologies like C# or Java. They have done a lot of work to optimize PHP and will continue to invest in optimizations instead of switching the language they use on their front ends.

Their dedicated web services are built in whatever technology the developer building the service is most comfortable with. Thus far they have dedicated services in C++, Erlang (see Eugene Letuchy’s post on Facebook Chat’s usage of Erlang), Python and Java. They have also built a number of tools that act as building blocks for creating new services so their developers do not reinvent the wheel. These building blocks include tools like Thrift; a cross-language RPC framework that allows you to communicate using serialized objects in multiple languages C++, Java, Python, PHP, Ruby, Erlang, Perl, Haskell, C#, Cocoa, Smalltalk, and OCaml).There is Scribe which is a system for aggregating server logs in real-time and publishing them to a centralized location. There are also custom internal tools for centrally managing configuration of services, as well as monitoring and alerting so that each new service doesn’t have to figure out how to solve these problems but can instead plug into what they have already.

An example of a dedicated internal service at Facebook is MultiFeed. This is the service which takes the tens of thousands of updates from a user’s friends and whittles them down to the forty five updates that are shown on the home page when a user logs in. When a user publishes an update it is both written to MySQL and then published to a their memcache cluster using Scribe. There is a cache of the fifty most recent items each user has performed always stored in memory. When a user hits the home page, an aggregator takes the list of the user’s friends and then queries a bunch of leaf nodes for activities their friends have performed. Each leaf node does some filtering before returning data to the aggregator which in turn does some filtering of its own before returning a list of story IDs to the calling application. The application then looks up the actual items from memcache given their story IDs and then renders the home page.

Multifeed is a custom distributed system which takes the tens of thousands of updates from friends and picks the 45 that are either most relevant or most recent. Bob updates status to DB and then Scribe pushes this to a Multifeed server which is acache of the most recent 50 items the user has performed. This caches data for every user in the system. When a user shows up, the user hits an aggregator which takes the list of friends and then each leaf node does some filtering then gets back to the aggregator which then does some filtering and returns story IDs. Then the aggregator fills out the data from memcache given the story ID.

Facebook heavily uses memcached which is an Open Source distributed, in-memory hash table they use to keep load off of their MySQL databases. They have a love/hate relationship with memcache. On the on hand it is reliable, has low latency and can handle an extremely high service rate. On the other hand data is easy to corrupt, the data model is limited and it is an inefficient way to store small items. They have made several customizations to memcache such as porting it to 64-bit, moving from TCP to UDP, optimizing the network stack and adding multithreading support. Their changes have improved the throughput of memcache by a factor of five compared to when they first started using it.

Once they got memcached throughput that high, they hit another scaling bottleneck in that they started flooding their network switches due to putting too much data on the network. This problem was addressed by using a throttling approach where web servers request data piecemeal from the caches instead of in one huge request. They performed a number of experiments comparing batch request size to memcached cluster size and came away with the conclusion that requesting items in batches of 50 was the most optimal solution for their system. This throttling also applies at the user facing application layer. For example, instead of requesting all forty five items at once when rendering the homepage, a subset of the data is requested and then only when the users scrolls down below the fold is the rest of the data fetched [Ed. Note – hey, I’ve noticed that when using Facebook]. They try to keep the memcache clusters small so that web servers don’t have to hit lots of cache servers to satisfy a request. To ensure this locality, data is partitioned vertically by object type to ensure that all instances of a particular type of object reside on a small set of cache servers. Since this creates single points of failure, objects in the cache are sometimes replicated. The term replicated is used loosely to describe the process of the application layer storing objects on multiple cache servers to simulate database style replication. When asked how they deal with data inconsistency issues, the response was that for the most part they limit the object types they “replicate” in this manner then live with the data inconsistency issues.

MySQL was initially chosen as their database of choice because it is free, simple, fast and reliable. All told, they have about 6000 server years of runtime experience without data loss or corruption due to software. However like most megascale services they often end up using MySQL as a glorified key<->value store as opposed to taking advantage of the relational features of the database. When asked why bother using a relational database, the response is that the database management features such as data replication and administration are still valuable even if you aren’t using foreign keys and other relational features. One interesting challenge is that they often use “data driven schemas” to enable programmers to add new types to the database without requiring schema changes. This approach sounds similar to what Bret Taylor described in his post How FriendFeed uses MySQL to store schema-less data. The challenge with this approach is that you end up with key<->value pairs stuffed in some column in the database where you can’t even efficiently query because you can’t index the fields you care about.

One of their biggest unsolved problems is coming up with a good story for being a geographically distributed service. Right now they have a situation where they have West Coast and East Cost data centers where they use the West Coast DCs for read & write operations while the East Coast data centers are read-only. This means a user in New York hits the East Coast DC when reading their news feed but hits a DC in California when publishing a status update. To keep the data in sync across data centers they use MySQL replication and have hacked the replication streams to also propagate memcached cache invalidation commands as well. This ~~clever solution~~ hack has been previously described in the Facebook engineering blog post by Jason Sobel entitled Keeping Up. There are some tricky problems they have to solve such as users often wanting the behavior to seem instantaneous despite the replication lag (e.g. I often change my status then want to check to see what it looks like on my profile/feed). There is also the open question on whether this solution is workable at a global scale (i.e. if Facebook ever adds data centers outside of the United States).

The company has released or contributes to a ton of Open Source software. Besides aforementioned technologies like Thrift, Scribe and contributions to memcached there are also a number of other interesting Facebook originated projects like Cassandra and Tornado. They also contribute quite heavily to Hive which is a data warehousing solution built on top of Hadoop.

There are three key mantras that describe Facebook’s engineering culture

Move Fast and Break Things – The company encourages people to take risks and innovate even at the risk of jeopardizing the stability of the site. It is important to them to not become risk averse because this is often the downfall of companies as they grow.
Huge Impact with Small Teams – Most features and underlying systems on the site are built by teams of one to three people. They have a metric of users per developers that they are particularly proud. They claim that at Facebook they have 1 developer for every 1.2 million users which is better than most industry leaders including Google (1 developer per 190,000 users), Amazon (1 developer per 96,000 users) and Microsoft (1 developer per 75,000). [Ed. Note – not to be defensive but this metric seems weird to me and it wasn’t clear how it is calculated. ]
Be Bold and Innovative – self explanatory

There were a number of interesting Q & A questions asked and answered but I ran out of juice in my laptop so didn’t capture them. The answers I do remember are sprinkled in the notes above.

I found it very interesting to learn about the architecture behind MultiFeed as well as the various scaling challenges and unsolved problems that Facebook still faces. I'd say there is something like 80% overlap in the way we've approached solving problems and where we differ our approaches seem more philosophical than architectural. For example, when our service hit similar issues around network switch saturation due to high throughput from our in-memory caches we solved the problem in a simpler way than Facebook's throttled back-off approach although I have to admit their solution is clever. Our approach was to look for ways to reduce the amount of data we put out on the network instead Facebook’s approach of staggering requests to reduce the peak load on the system.

If you’re interested in working on or building systems similar to the ones described above for the 500 million users who utilize Windows Live services. Send me a resume, we’re always looking for good developers.

Now Playing: Yo Gotti - 5 Star (Remix) [feat. Gucci Mane, Trina & Nickie Minaj]

RSS Bandit + Facebook = RSS Bandit 1.9.0.972

2009-10-13T09:01:45.0366548-07:00

Over the weekend, Torsten and I shipped a new release of RSS Bandit. Besides bug fixes there is one key new feature in the release, the ability to view and comment on your Facebook news feed. The flow for adding Facebook to the application is as follows. Go to the File->Synchronize Feeds menu option then select "Facebook"

then go through the Facebook Connect authorization flow including optionally signing in and granting the application permission to view your news feed

This creates a new feed source containing your Facebook news feed complete with inline comments as shown below

You can download the new release from here. More details about the bug fixes in the release are in the official RSS Bandit blog post on the release.

The primary purpose of this release (codenamed Colossus) was to bring stability to the code base before we made radical changes. With this release out of the way, we will start working on the Gambit release right away. The purpose of the next release is primarily to make RSS Bandit a more modern application that looks like it belongs on Windows 7 and Windows Vista instead of harkening back to the Office 2003 look of yesteryear.

You can see our plans for updating the RSS Bandit user interface in my blog post and prototype screenshots of the RSS Bandit ribbon. We will also support new features of Windows 7 such as jump lists. As usual, comments and feedback are welcome.

Now Playing: Dashboard Confessional - Stolen

Duct Tape Programmers and the Culture of Complexity in Software Projects

2009-09-27T09:00:30.767396-07:00

Last week Joel Spolsky wrote a blog post entitled The Duct Tape Programmer where he praises developers who favor simple programming practices to complex ones. This blog post strongly resonated with me and made me recall some related thoughts on complexity and solving problems in software projects. Some key excerpts from his which I'll use as a jumping off point are below

Jamie Zawinski is what I would call a duct-tape programmer. And I say that with a great deal of respect. He is the kind of programmer who is hard at work building the future, and making useful things so that people can do stuff.
...
Duct tape programmers are pragmatic. Zawinski popularized Richard Gabriel’s precept of Worse is Better. A 50%-good solution that people actually have solves more problems and survives longer than a 99% solution that nobody has because it’s in your lab where you’re endlessly polishing the damn thing. Shipping is a feature. A really important feature. Your product must have it.

One principle duct tape programmers understand well is that any kind of coding technique that’s even slightly complicated is going to doom your project. Duct tape programmers tend to avoid C++, templates, multiple inheritance, multithreading, COM, CORBA, and a host of other technologies that are all totally reasonable, when you think long and hard about them, but are, honestly, just a little bit too hard for the human brain.

The urge the reduce the complexity of the tools used to solve software problems is one that every software developer should share. However even more important is reducing the complexity of the actual solutions that are delivered to your customers at the end of the day. End users can't tell if you used complicated C++ techniques like template metaprogramming and mixins to build the application. They can tell when your application fails to solve their actual problems in a straightforward way or is so late to ship due to project delays that they lose interest in waiting for you to solve their problems.

There are many famous and everyday examples of this culture of complexity in software projects which are eventually trumped by solutions that solve 80% of the problem in a simple way. My favorite example is contrasting the World Wide Web invented by Tim Berners-Lee with Project Xanadu as envisioned by Ted Nelson. Today the WWW is used by over a billion people to enrich their lives in myriad ways on a daily basis and has created hundreds of billions dollars in value by minting an entire new industry. Project Xanadu is a sad footnote spoken about in hushed tones by fans of hypertext who bewail the success of the Web and how it has forced us to settle for less (i.e. Worse Is Better).

If you aren't familiar with Project Xanadu you can think of it as a networked system of hyperlinked documents and media just like the WWW which had to satisfy the following seventeen rules

Every Xanadu server is uniquely and securely identified.
Every Xanadu server can be operated independently or in a network.
Every user is uniquely and securely identified.
Every user can search, retrieve, create and store documents.
Every document can consist of any number of parts each of which may be of any data type.
Every document can contain links of any type including virtual copies ("transclusions") to any other document in the system accessible to its owner.
Links are visible and can be followed from all endpoints.
Permission to link to a document is explicitly granted by the act of publication.
Every document can contain a royalty mechanism at any desired degree of granularity to ensure payment on any portion accessed, including virtual copies ("transclusions") of all or part of the document.
Every document is uniquely and securely identified.
Every document can have secure access controls.
Every document can be rapidly searched, stored and retrieved without user knowledge of where it is physically stored.
Every document is automatically moved to physical storage appropriate to its frequency of access from any given location.
Every document is automatically stored redundantly to maintain availability even in case of a disaster.
Every Xanadu service provider can charge their users at any rate they choose for the storage, retrieval and publishing of documents.
Every transaction is secure and auditable only by the parties to that transaction.
The Xanadu client-server communication protocol is an openly published standard. Third-party software development and integration is encouraged.

Reading this list is like going through a list of places where World Wide Web fails. Rule #14 which implies every document on the network is redundantly backed up in disparate locations so they can always be is something the WWW doesn't do today which is why we have broken links and 404s all the time. Rule #9 implies that not only is copyright respected and tracked throughout the system but there is even a micropayment platform built in. All the discussions on micropayments saving newspapers would be moot if Project Xanadu ruled the world since it would have existed from day one. Rule #16 on transactions being secure and auditable sounds like Nirvana in today's world of botnets, malware and phishing scams which plague the Web.

Yet despite the fact that the forty year old Project Xanadu is a more compelling vision than were we are today it failed and Tim Berners-Lee's World Wide Web succeeded. In practical terms, Project Xanadu was trying to solve too many complex problems in a v1 product. In contrast, Tim Berners-Lee focused on the most valuable problems to solve for end users which was sharing documents and media with anyone on the Internet and punted on a bunch of the hard problems that would require a more controlled and tightly coupled network as well as a ton of more code. Tim Berners-Lee solved less than half the problems Project Xanadu set out to solve but has changed the world immeasurably for billions of people by providing simple solutions to complex problems and running away from trying to create complex solutions to complex problems.

The bottom line is that a lot of the time it's OK to create a solution that solves 80% of the problem. Always remember that shipping is a feature.

Now Playing: Drake, Kanye West, Lil Wayne & Eminem - Forever

The Brilliance of Twitter's Suggested User List

2009-09-27T08:12:24.6608672-07:00

Every week there seems to be some new A-list blogger criticizing Twitter's Suggested User's List which is a selection of celebrities and brands that are suggested to new Twitter users as people the user might like to follow. This week it's Robert Scoble with You’re not on Twitter’s suggested user list but you are in good company that points out a number of interesting celebrities and brands that aren't on the list. Last week Dave Winer asked The SUL as a tool to control news?

I've had my issues with the SUL mainly from the perspective of how it ends up presenting Twitter to new users. When my wife joined Twitter I'd have loved it if the service had used integration with Facebook, Windows Live, MySpace, etc to suggest people who she already knew who were on Twitter. Instead the service prioritized pitching that she follow Shaquille O'Neal, Dell Outlet stores, NBC's Today Show and Jessica Simpson's kid sister. To find me on Twitter, my wife had to ask me for my Twitter handle in person. I felt like we were back in the dark ages of social networking.

In retrospect, not doing what I preferred them to do shows a lot of insight. It prevents the site from being viewed as yet another service where you have a duplicated social graph and thus has to compete head to head with the Facebooks and MySpaces of the world. Instead it pitches Twitter as a sort of user friendly RSS reader where you connect with your favorite celebrities and brands instead of another place where you get status updates from people who you're already getting status updates from in Facebook.

Brilliant.

Now Playing: Jay Sean - Down (feat. Lil Wayne)