hueniverse

You, Me, and Node @WalmartLabs

2012-01-04T23:04:20Z

Another adventure begins.

Two months ago I’ve joined @WalmartLabs to lead the mobile web services team. Surprised? I was. After working for one of the largest web companies in the world, all I wanted to do was go to a startup. That’s not exactly right; I wanted to be part of a tiny team with a big mission, a place where the size of the challenge is matched by the freedom and resources to address it. Oh, and a lot of node.js!

I am excited to share this and tell you all about it, especially on the heels of this morning announcement of the acquisition of Small Society. But I’m not going to lie to you: I have an agenda and I am trying to recruit you. If you are contemplating a career move and I “had you at node”, feel free to jump right to very end of this post to find more about the team we’re building.

Rethinking Retail Through Mobile

It’s all about size, especially when it comes to consumer retail. @WalmartLabs is part of Walmart’s Global E-Commerce group and is focused on reinventing retail through technology. This is among the most important strategic investments Walmart has made in recent years and it goes deep and wide. This is not simply about making online shopping better, but about fundamentally changing retail by creating a new, continuous shopping experience across technologies and venues.

Our team is focused specifically on executing this vision through mobile. Unlike most other mobile retail applications, our focus isn’t limited to buying stuff on your phone or tablet. We are looking at the full range of possibilities when you have a powerful mobile device in your hand, both inside and outside the store. In addition, we are looking at a much wider set of products, from electronics to pharmacy and all the way to fresh groceries.

The best part about building a team and talking to potential new hires are the new product ideas that come up when you start to think about what’s possible: An app that sorts your shopping list based on aisle location as you walk into a store; the brick-and-mortar version of ‘60% of your friends bought this brand of toothpaste’ when trying to decide which to buy; real-time checkout, showing you the total dollar amount including tax of the items in your cart; shopping analytics across stores and online purchases producing a new and unmatched dataset about your lifestyle and health as the foundation of new tools to improve your life. And this is just from my most recent conversation.

From my first meeting with the team, I found myself thinking about new ways to change retail. I’ve never given much thought to that, other than noticing a feature I liked on Amazon or a time-saving service at a store. All of a sudden, I’ve started noticing all the small things we can change today – the obvious ideas just lying on the floor – if only I could get those apps in the hand of enough people. And that’s where Walmart offers a unique opportunity, with hundreds of millions of loyal customers worldwide eager to give your new idea a try.

Working for Walmart? You Can’t Be Serious!

Now let’s talk about the big blue elephant in the room: working for Walmart. I’m in no way trying to change how you view Walmart; only to share with you the process I went through.

To put it bluntly, I had a gag reflex reaction to the idea of working for Walmart. It’s hard for me to pinpoint exactly what my reaction was based on. Walmart’s image, history, and impact on society played a role, but it was mostly my recent experience working for a large web company and past experience working for Citibank, the largest bank in the world, that turned me off. Short of working for the Department of Defense or the Chinese Army, nothing gets bigger than Walmart.

This was not the employer of my dreams – far from it! – but the opportunity itself, and more importantly the caliber of the team, convinced me to take a deeper look. I spent hours reading up on Walmart, the good and the bad. Given the size of the company, there was plenty of both. I was surprised at how much I didn’t know about the company.

I was mostly blown away by the numbers, of which two really stuck with me:

in the United States, 140 million people visit a Walmart store every week, and
Walmart is responsible for 35% of all groceries sold.

I remembered an article I read a few years ago about Adam Werbach, once the youngest president of the Sierra Club, and his decision to join Walmart and the backstory of that decision. I also recalled the Stonyfield story in Food Inc. and the impact Walmart had on the landscape of organic food (price, availability, and commoditization) once they decided to carry it in their stores.

I’m not trying to draw any comparisons between these stories and my personal decision, other than to point out that when you get to point the “Force” (or “Death Star”, depending on your perspective) at a target, big things happen.

At the end, we all make compromises when we work for any corporation, especially those large enough to have real impact. Walmart’s size translates directly into a noticeable impact on society but without getting too philosophical, so does Google’s, Apple’s, and News Corp’s to name a few. I have major grievances with all these corporations, but I do recognize that if I was trying to recruit you to work there, I would not be spending this much time to get you to look past the brand’s reputation. I respect the fact that for some it will always be “you lost me at Walmart”.

What Do I Want to Do When I Grow Up?

Last year I wrote about my decision to stay at Yahoo! and work on Sled, a virtual startup within Yahoo! building a collaborative list-making tool. The list included products, culture, manager, team, and work/life balance, and these are still the main factors that matter to me. When the time came to move on to another opportunity, I’ve asked myself a two questions: what is it I’m most excited about these days, and who is it I am most interested in working with?

My passion and focus the past four years has been around APIs: frameworks, design, advocacy, and lot of security (you know, all that OAuth stuff). But my true joy over that period is a lot more specific: node.js. Working with node for over a year in a full-time capacity has spoiled me and for the first time in a while, added a specific technology to my job search criteria.

In one sentence, my dream job was leading the development of a new, world-class, open-sourced, node-based, OAuth-protected, developer-focused, API server framework.

I wanted to take all my experience and ideas from the past few years and apply them to one awesome project. But that wasn’t enough (is it ever?). It had to be real. I didn’t want another academic exercise or working in a vacuum. I also had no energy to constantly fight my employer to let me do what I was hired for. I wanted to build something that will get used and abused as soon as it was forming, ideally by millions of users.

I wasn’t expecting to find it but I actually ended up with two such opportunities – both amazing. One was an early stage startup and the other, Walmart. The Walmart opportunity was a surprise. One of the people I really wanted to work with was Dion Almaer. We have not worked together before but we did cross paths over the years and I’ve been following Dion’s and Ben Galbraith’s work with interest. Turned out that earlier this year they sold their company to Walmart and took over mobile engineering, helping to create @WalmartLabs.

Ben and Dion’s story on how they got to Walmart is pretty interesting by itself but even more telling about the team and the kind of opportunities it offers. At the end it wasn’t an easy decision and it was very close. I’m happy to report that a 2 months into this, I’m super excited about my decision and it seems to be the prevailing attitude shared by the rest of the brilliant people I get to work with which include Kevin Decker leading the mobile web team, Stefan Li leading the Android team, Brandon Sneed leading the iOS team, and the new folks from Small Society to mention just a few.

Let’s Talk About You!

Now that you know how I ended up at Walmart and got the big picture of what we are trying to do, we can talk about what it is we are actually doing and how you might fit in.

To expand a bit on my one sentence job description, the mobile services teams is in charge of the backend serving all of our mobile applications. Today these include iPhone, iPad, Android, and mobile web. We are mainly focused on the US market but will expand internationally in the coming months. We have a legacy JAVA system in place which we are going to replace over the next few months with a brand new node solution, as well as develop new services directly in node.

Our main focus is building a world-class web services operation which includes building the best API server framework possible. Think of it as Express for APIs, with the typical requirements for speed and flexibility, but designed in a way that enables building an entire API service in days and focus primarily on business logic instead of facilities (e.g. query and payload validation, smart routing, dynamic documentation, OAuth support). And of course, we are going to build the core of this framework as open source from the very beginning.

In addition, we are looking to build and contribute to projects aimed at making mobile services better, including smart caching (we are currently evaluating Redis and other new technologies as the foundation of such facilities), and dynamic repositioning of HTML rendering services between the back-end and front-end. Also, in alignment with our ‘Labs’ name, we experiment with new technologies and architectures aimed at improving mobile performance end-to-end.

The current services team is about four people and we are looking to add a few more over the next few weeks. The new roles will be 100% node, working on the new framework and services. Unlike most other “node jobs”, this is really all about node, not a trick to get smart people excited about yet another JavaScript gig.

If you noticed, I haven’t wasted your time with stories about how, despite the size of the company, it still “feels just like a startup”. After all, this is what every large company tries to sell these days. Instead, I invite you to come and check us out, talk to the team and form your own opinion on how lean and mean we really are. Most of our team joined us from recent startup experiences and they care a lot about that culture and energy. Our team is fairly distributed with the main office located in a new building in San Bruno, CA. The iOS team is based in Portland, OR, and we have quite a few team members working remotely.

We are looking for brilliant engineers passionate about new technologies in general and node.js in particular. We are not going to bore you with a useless list of experience and requirements. If you think you have what it takes to get the job done, we want to talk. We promise not to bring you in for an endless parade of grilling interviews or ask you to write code on a whiteboard. We much rather check out your Github account and have a lively debate about technologies, software design, and your obligatory bizarre geek hobbies (we’ve got our own to share).

I will be writing about our progress and the new framework over the next few months. This promises to be an exciting ride. If you want to learn more, drop me a line.

Sled, Yahoo!, and Moving On

2012-01-04T23:36:13Z

As is often the case when working on a startup, virtual or not, things don’t work out according to plan. I had the privilege to spend the past year working on Sled, from product inception to execution. At the same time we were launching Sled, I experienced some significant management changes at Yahoo! which eventually led to shutting it down.

To be clear, the decision to discontinue Sled was mine alone, but it wasn’t made in a vacuum. The change in management brought with it a change in attitude towards the project in both form and substance. My options were to stay and fight for it, or shut it down and move on. It was pretty upsetting, as these events usually are, especially given how active the list-making / light collaboration space is and the huge potential there. After fighting the organization for three years, I decided to move on and left the company shortly after.

This was the official announcement posted at the time on the now defunct Sled blog:

Over the past few months, I had the pleasure of being part of an exciting experiment at Yahoo! called Sled.

Sled was a collaborative list making tool with a strong focus on life events and collaboration between friends and family. Planning a party, a house move, getting ready for a new baby, planning a trip, organizing a junior soccer league, or preparing for a marathon, are some of the areas Sled was focused on. We also found it really useful for building Sled itself, working in a small team, keeping track of issues and assignments.

There is a lot we don’t know about how to make our daily lives more productive and organized, and how to collaborate better. What we do know is that the wide range of tools and services available to us are, generally speaking, not very helpful. Everything is either too limited or too complicated. Usually too complicated. We know what doesn’t work, but figuring out what does requires experimentation.

Sled was developed as a virtual startup at Yahoo!, using open technologies and the freedom to experiment. As is often the case with startups and experiments, we have been constantly evaluating the product and its fit within our existing and future roadmap, and have made the decision to discontinue the project in its current form (shutting down sled.com at the end of the week).

But our story doesn’t end there.

We built Sled on an entirely open stack, using open source tools (Node.js, MongoDB, Express, Socket.IO, Jade) and open standards (JS, HTML5, OAuth 2.0). We have relied and benefit from a vibrant community of amazing developers and we want to give something back. One weak area for the community is the availability of fully-baked, showcase applications written in Node.js.

Instead of following the typical industry path of discontinued products, we have decided in the experimental spirit of Sled to release the entire project under an open source license using a new name: Postmile. This means anyone will be able to grab the code and run the entire service, back and front, exactly as it was hosted on sled.com (including the soon to be open sourced iPhone app we never got to release).

We hope you will find Postmile helpful, fun, and an insightful resource for the kind of projects you can build with Node.js today. Check it out at http://j.mp/postmile. You can use Postmile as a back-end API server for building new list-based products (to-do apps, simple project management tools, bug tracking), as a list making tool for your friends or your company, or just as a useful example to grab code from.

The best part about working on Sled was node. I got the rare opportunity to spend a year working full time on node and be part of the node community. The experience was so fantastic that I made working with node a central theme of my job search and I’m happy to report that it worked out quite nicely. But that’s for another post.

Is the Party Winding Down at Facebook?

2011-12-12T17:03:33Z

A picture started to emerge from casual conversations I’ve had over the past few weeks with friends working at Facebook. I have noticed how Facebook engineers are using a different, more restrained vocabulary to describe their jobs. What once was ‘amazing’ is now ‘challenging’, ‘exciting technology’ turned to ‘learning a lot’, and ‘having fun’ toned down to ‘still engaged’. They are all very ‘content’.

This might be purely anecdotal but I have a feeling it is not. I’ve shared this insight with a few other friends outside of Facebook and they too started noticing the pattern within their own circle of Facebook connections. Unlike a year ago, these days I can’t point to a single person I know working at Facebook who is oozing excitement and energy. They are still very much engaged and are busy as ever, but they are less interested in talking about work.

While this kind of corporate evolution is expected, it is a bit early for Facebook to make the transition from a hyper startup to a business-as-usual large company culture. If this continues, the looming IPO is going to include a noticeable loss of talent with what feels like a new holding pattern within the company of people waiting to fully vest or cash out.

Netflix Forcing the Issue Too Soon

2011-12-13T04:22:30Z

(A note to my long-time readers, I’m planning on expanding this blog to include opinions about current technology trends and news beyond my usual fare of standards, open web, and engineering posts.)

This morning I logged into my Netflix account and changed my plan from 3 DVDs + Streaming to DVDs Only. Despite the excellent analysis by many about Netflix’s reasons for the recent changes, the fact is that today the company is making $95.88 less annually from me than they did the day before. That’s a lot of money to lose from a single, loyal customer.

I love the unlimited streaming service, but given its current selection of movies, I use it mostly to catch up with TV shows I missed or skipped. We do this mostly during the summer TV hiatus. In other words, the current streaming catalog is great when there is nothing else to watch and you are willing to compromise with lesser, somewhat stale, content.

I’m probably going to get a streaming service back in June when the 2012 season is over. But at that point I will no longer default to Netflix because I have no incentive to do so. It doesn’t save me money or add any nice features to my core DVD-by-mail service (today, I can see what in my queue is available on-demand and consume it that way). I will definitely consider Netflix as an option, but if my cable provider, Apple TV, or XBOX, offer me an easier alternative with the right content, I’m probably going to take the lazy way out.

Strategically, this was the right move. Tactically, it’s a big risk to take given the blow it can serve to their momentum. Everything communicated by the company makes perfect sense to me. The problem is that by pushing the issue now, they have forced me to think about their separate services when their weaker offering is the one they are betting their future on.

The Unauthorized Node Knockout #2 Awards

2011-09-04T07:32:09Z

I’ve spend the past week participating as a judge in the 2nd Node Knockout competition – a 48 hours worldwide hackathon using node.js. The event included 720 contestants organized in 294 teams and resulted in 178 entries submitted for review. Overall, a fantastic event and a testament to the awesomeness that is the node community.

The competition includes prizes for the best hack in the following categories: fun or utility, design, innovation, completeness, popularity, and overall, as well as overall for a solo participant. While judging is still on-going, and while many of the top entries do deserve to be there, I found the categories to be a bit uninspiring.

Also, as the only crazy person to judge every single entry (including some that dropped out in the process), I wanted to highlight some entries that might not get the attention they deserve. The following are the nominees and winners in my unauthorized awards.

Most Geek-elicious

And the winner is: Node Defense

Biggest Time Suck

And the winner is: Metris

Most Practical

And the winner is: nide

Best Art Direction

And the winner is: Gemini Command

Most Original Classic

And the winner is: Pong

Most Bizzare

And the winner is: Boys over Flowers

Biggest WOW Moment

And the winner is: LocalNode

Congratulations to everyone who participated. I highly recommend you to check out every single entry listed on this page. This is the best of the best – and I’ve seen them all! Also, a big shout out to Gerad Suyderhoud and Visnu Pitiyanuvath for the amazing job they have done organizing the event.

Can’t wait for next year!

Twitter Accounts

2011-08-02T02:22:22Z

Just a quick update. I’ve split my twitter account into two. Follow @hueniverse for blog updates and other information about the technical topics I discuss on this blog. Follow @eranhammer for my personal brain farts.

A Farmer Walks into Facebook

2011-07-19T06:57:15Z

A few weeks ago I went to visit some friends at Facebook’s headquarter. We had an interesting chat about OAuth and other geek topics. As is common these days, the conversation drifted to my recent adventures in farming. I was describing my setup, the chickens, ducks, geese, and pigs I’ve got running around, and then mentioned my three emus all named Kevin.

Someone who was listening in from a nearby cube stood up and asked, “When did they add emus?”

Get it? At Facebook everyone’s a farmer.

OAuth 1.0 Blog Cleanup

2011-07-16T06:33:04Z

As I’m getting ready to finish work on OAuth 2.0 and add new content to this site, I decided it was time to finish the OAuth 1.0 chapter of this site. I’ve finally cleaned up the OAuth 1.0 guide and other pages. The guide is now updated to reflect RFC 5849 as well as some bug fixes in the scripts used to generate the signature base string tutorial. If you are linking to this site for OAuth resources, please link to the OAuth page.

Introducing Sled

2011-07-14T22:17:55Z

I’ve been obsessed with project management and personal productivity for a almost two decades. My experience ranges from tiny lists to gigantic project plans with hundreds of people and resources. In the past I’ve been a certified PMP and managed large engineering teams. What I’ve learned above all, is that we tend to overcomplicate everything.

Four years ago my startup Nouncer failed for many reasons, none of which had much to do with the product itself. Looking at where Twitter is now and how it evolved, it is a clear validation of my original vision. But even if I had gotten passed the challenges that doomed Nouncer, I still think it would have failed. It was just too complicated, too soon.

I’ve long considered Twitter’s biggest asset to be its 140 character limit. It completely democratized personal expression by making everyone as expressive and articulate. It also helped people communicate more by making their content small enough for casual, constant consumption.

A year ago I started thinking about applying this philosophy – empowerment through restrictions – to project management. I’ve started thinking about enterprise-scale problems and what a restrictive tool might look like. But no longer working on large scale enterprise projects, my attention shifted to personal productivity and “home projects”, and so Sled was born.

Sled is an experiment.

There is a lot we don’t know about how to make our daily lives more productive and organized, and how to collaborate better. What we do know is that the wide range of tools and services available to us are, generally speaking, not very helpful. Everything is either too limited or too complicated. Usually too complicated. We know what doesn’t work, but figuring out what does requires experimentation.

Sled is a collaborative list making tool with a strong focus on life events and collaboration between friends and family. Planning a party, a move, getting ready for a new baby, planning a trip, organizing a junior soccer league, or preparing for a marathon, are some of the efforts I hope Sled will be useful for.

The idea is to start with a really solid tool for making lists. Then add collaboration with a few people, a little bit of social features, and later a crowd-sourced knowledge base of lists and recommendations. For the past few months we’ve been focusing on list making and the result is a simple tool to create lists alone or with a small group of friends. Sled is still in its very early stages, but it is usable and (we hope) fun.

I’m not planning on writing much about Sled on this blog. Instead, I will focus on sharing my engineering experience building the product. If you want to keep up to date with Sled, follow us on Twitter or Facebook, or check out the Sled blog.

And if you want to check it out, sign-in with your Yahoo!, Twitter, or Facebook account and use invite code: hueniverse.

Node.js: Express, Socket.io, and everything LearnBoost

2011-07-14T22:20:56Z

This post is part of a series of articles about my recent experience building Sled using Node.js.

Express It

There wasn’t much of selection 6 months ago when I started coding Sled when it came to Node frameworks. Node itself provides very little. You create an HTTP server and get a callback when a new HTTP request comes in. Modern web applications require session management, request routing, and view rendering at the very least. These functions are provided by frameworks.

At the time, the only two popular frameworks were Connect and Express. Connect provides a basic middleware framework with some built-in facilities such as a static file server, route management, cookie parser, form-encoded and JSON-encoded body parser, and logging. Express, built on top of Connect, adds a much more robust routing facilities, view rendering and other goodies. I’m not doing these frameworks justice with this descriptions.

I use Express extensively and it is fantastic.

Express’ biggest asset is its maintainer, TJ Holowaychuk. I have seen first-hand how Express grew and matured into a stable and powerful framework. Express made it easy for me to write the application server used for login, registration, account management, and other static assert (developer pages, about) with very little extra effort. If you are going to develop a traditional web application in Node, you should probably start with Express.

I plan to open source some of the additional functionality I’ve added on top of Express to validate request bodies and query parameters, a flexible authentication configuration facility for routes, and a light layer to make it easier to building API servers. Given my focus on OAuth, I’m going to share my OAuth + Express experience in a followup post.

Express really shines when you combine it with Jade, another brilliant brainchild of Mr. Holowaychuk – a simple templating language for HTML which is easy to learn, easy to read, and unlike all the rest, doesn’t suck. We had to restrain ourselves from converting some static HTML files into Jade because once we started using it, we didn’t want to read actual HTML ever again.

Socket.io is Magic

Really!

I am willing to bet that a large percent of those taking a look at Node for the first time are doing it because of a Socket.io -powered demo they have seen. Socket.io provides a trivial-to-use server and client library for making real-time, streaming updates between a web server and browser client. It makes building cool real-time games a matter of hours, and it works on every crappy browser known using the best available option from Flash to WebSockets.

We use Socket.io to power Sled’s real-time features which include live updates to your shared list as changes are made. To put this in perspective, it took us one day to add real-time streaming updates to the API server, and another day or two to add it to the client. So in three days we got full, real-time updates going between multiple browsers. What used to be months of work when Google Docs was first introduced, is now trivial with Socket.io.

So yeah, it’s magic.

Socket.io comes from Guillermo Rauch and the team of superstars at LearnBoost. There are days when I have to wonder if these guys get anything done for their own startup, given the amazing open source projects they push out on a weekly basis. I’m bummed that I’m not the target audience for their product because looking at the technology it must be amazing. I consider them part of the Sled team, given just how much of their code we are using.

In fact, I appreciate their work so much, I’d like to get a small fundraiser going to send these guys to dinner or whatever else they feel like doing for fun. I hope you join me in showing our appreciation. Without their continued work and dedication, Node would be an amazing platform that is just too raw to use.

Thanks to Blaine Cook for the idea.

Node.js: From Couch to Mongo

2011-06-30T06:24:31Z

This post is part of a series of articles about my recent experience building Sled using Node.js.

I started with CouchDB and ended with MongoDB.

Working with CouchDB was fantastic. It took no time to learn the REST API and jump right into building the application. I had the first version of the base API ready in 2 days, including full database integration. Couch is a document store using JSON as the document format. Combined with an HTTP REST API, it makes Couch an ideal fit for Node. That’s exactly why I picked it.

I dismissed MySQL from the very beginning for two reasons. First, I had no idea what my schema would look like, and knew I was going to change it a lot over time, as well as allow unstructured data. Second, MySQL is a block database in nature (e.g. database deadlocks, atomic actions, etc.) and while you can use it with Node, it really wasn’t designed for this kind of environment. Also, at the time, there was no quality driver for Node. Oh, and I really wanted to play with a NoSQL database.

I started talking directly to the database but within a few hours switched to use Cloudhead‘s excellent cradle module. The module provides a light layer for managing the HTTP client calls, error handling, simple caching, and common macros for performing multiple database requests in a single function call. It is a very light abstraction layer (and it doesn’t abstract too much either). This worked very well for a few months. I didn’t use cradle’s built-in cache because of the expected size of my database and my constant tweaking of data manually.

Once we started stress testing the server, we hit one of the most common problems with Node’s asynchronous model: the inability to control the execution order of events. We have a simple document which includes a name, place, time, and date. The properties can be modified by multiple people, and if two people change the same property, the last update wins. But when two people change different properties (one the title and the other the place), there should not be any conflict.

To update a document in Couch, you must have its current version. Typically this means going to the database, grabbing the latest version, changing it, and saving it back. The problem is, this simple process breaks into two callbacks, one for fetching the latest document, and another for saving the modified version. When two requests come in at the same time, Node will queue two database fetch requests (for the same document) and then will process each in order. Both requests will return the same document, but after the first changes the document, the second update fails.

I came up with a few ways to address this race condition (I’m sure there are more):

Retry as many times as it is likely to get multiple updates to the same document at the same time. So if we expect 5 people to update the same list at the same time, we should retry at least 5 times.
Use a local cache to store the latest version of recently modified documents and a queue of pending updates. When the first request comes in, the change is added to the queue and the latest document is requested from the database. When the latest document is obtained, the update is applied to both the cached copy and the database. When the second request comes in, the cache can be in one of two states: have the latest document in memory, or pending. If the document is available, it is updated and saved. If not, the second update gets added to the queue. When the document is received, both updates are applied at the same time.
Perform lazy database updates, batching together multiple updates into a single commit. This is performed as soon as there are updates, but it applies all the accumulated updates after it gets the latest copy.
Use another database with native support for partial updates.

#1 works but is butt-ugly. It hard-codes load expectations and makes it harder to identify actual database errors. #2 could work, but I just didn’t feel comfortable with this level of complexity for something as fundamental as database updates. #3 means requests coming in between updates will include data that can be multiple-generations old. So I went with #4 and switched to Mongo.

The good news is that converting the application from Couch to Mongo took two days. I used the powerful native Mongo driver from Christian Amor Kvalheim. Overall the transition wasn’t bad, but it wasn’t trivial. The main difference is how the two databases store documents. Couch stores native JSON objects while Mongo has its own internal format which is similar but not identical to JSON. The biggest difference is Mongo’s support for a more complex set of numbers and native types.

Coming from Couch, I got pretty spoiled at working exclusively with simple JSON documents. JSON-in-JSON-out. If you can make Couch work for you, this alone will make your life much simpler. With Mongo, you have to decide how to handle numbers, dates, identifiers, and other data types. To make the conversion faster, I wrote a small layer on top of the native driver to normalize my JSON objects into and out of the database, converting ids to strings, and all numbers to simple JavaScript numbers.

I also had to add some restrictions on key names not to start with ‘$’ or include periods which are forbidden in Mongo (for a reason). We have an API allowing client developers to store a bit of data on the server for their own use. With Couch, we could let them dump a JSON object as-is (after some security sanity checks), but with Mongo, we had to filter out the forbidden keys and had to move to a key-value store instead.

On the other hand, the database itself became much simpler, removing the need to use Couch views (which meant one less place to manage code). I’m also loving the ability to manipulate arrays within a document with very specific instructions using Mongo’s powerful array update support.

Overall, if you need partial document updates as a basic database functionality, Mongo is a better fit, especially in Node. If you don’t, Couch is simpler and much faster to learn and use. I really love Couch and I wish I could use it. Mongo is pretty awesome too.

One major complaint for Mongo is the lack of decent management tools. Couch comes with the built-in Futon application which is a must-have for any new application development. If I had to use the Mongo command shell to get going in the first few days, it would have taken me twice as long to get going. The available tools for Mongo are awful. They are buggy, they crash, they corrupt data, and when they work, they require you to install Python, Ruby, or PHP on your server – something I refuse to do on my clean Node box.

Node.js: The Style of Non-Blocking

2011-06-30T06:25:41Z

This post is part of a series of articles about my recent experience building Sled using Node.js.

Node is all about non-blocking, asynchronous architecture. This means any activity taking a long time to finish, such as file access, network communication, and database operations, are requested and put aside until the results are ready and returned via a callback function. Instead of asking to read a file and waiting for the operating system to come back with a file handler or buffer, the a callback function is invoked when the operation is completed, freeing the server to handle additional requests.

What gives Node a bit of a negative reputation is how this architecture affects its style of programming and the difficulty some people are having in getting used to it. When I first started, I described this convoluted style of coding like scratching your right armpit with your left hand by twisting your left arm over the shoulder, behind the neck, and under the back of the right armpit. There are days when it still feels like that, but at least now my arms are much more flexible.

Consider the following code, used to fetch a database record and output the user name:

function getUser(id) {
    var user = db.query(id);
    return user;
}

console.log('Name: ' + getUser(432).name);

The function blocks until the database call is completed. This means the server is doing nothing but waiting until the function completes, ignoring other pending requests. In Node, the code is broken into two functions:

function getUser(id, callback) {
    db.query(id, callback);
}

function display(user) {
    console.log(user.name);
}

getUser(432, display);

However, using JavaScript anonymous functions, the code can be streamlined into:

function getUser(id, callback) {
    db.query(id, callback);
}

getUser(432, function (user) {
    console.log(user.name);
});

This nesting of function definitions inside function calls makes the code appear more linear and many find it easier to read. However, it can be tricky for new developers. For example:

getUser(432, function (user) {
    console.log(user.name);
});

console.log('Done');

is going to output ‘Done’ before it outputs the name because the name output waits for the database call to return, place the results on the event queue, and wait for the current executing code (outputting ‘Done’) to finish.

It takes time to get used to this style of coding, especially when your execution isn’t completely linear, but forks based on changing conditions. For example, we have an API call to add an item to a list. The call supports an optional parameter to insert the item at a specific position. Because we store each list item in its own database document, we keep the list order separately. This means, we sometimes make one database call, and sometimes many. At the conclusion, we perform additional processing and return the document id of the new list item.

This produces a very simple blocking code:

function add(list, title, position) {
    // Add new item to 'items' store
    var item = db.insert('items', { list: list, title: title });
    //  Set position if requested
    if (position !== null) {
        var sort = db.get('items.sort', list);
        addToListAt(sort, item.id, position);
        db.update('items.sort', sort);
    }
    // Perform final processing
    var result = { status: 'ok', time: (new Date()).getTime() };
    return result;
}

But not so simple non-blocking code:

function add(list, title, position, callback) {
    // Add new item to 'items' store
    db.insert('items', { list: list, title: title }, function (item) {
        //  Set position if requested
        if (position !== null) {
            db.get('items.sort', list, function (sort) {
                addToListAt(sort, item.id, position);
                db.update('items.sort', sort, function () {
                    finish(item.id);
                });
            });
        }
        else {
            finish(item.id);
        }
    });
    function finish(id) {
        // Perform final processing
        callback({ id: id, status: 'ok', time: (new Date()).getTime() });
    }
}

By embedding the callback function as an argument to the non-blocking function, and separating each logical part of the process into smaller sub-functions, you can code as fast with Node as with any other platform, but it takes some getting used to.

There are a few libraries out there that allow you to code using a blocking style, and turn that into non-blocking code automagically, but I found that to be counterproductive. There is tremendous value in seeing exactly what is happening at run-time and staying close to the execution flow.

Debugging non-blocking code is still very tricky. You can’t just put a breakpoint at the top of the function and step through it. What will happen is that you will break, complete the first request, and then move into the next event in the queue which is not likely to be the next step into your function. Instead, you need to put multiple breakpoints inside each nested function, and rely heavily on logging to analyze your code.

Like any other platform, Node has its own unique style and it can be off-putting to some. But it really doesn’t take long to get used to and once you embrace non-blocking development, the benefits in performance are significant. It also forces you to think hard about your code and architecture, to make sure you are not doing something stupid that will fail-whale you later.

Continue reading from Couch to Mongo.

6 Months with Node.js

2011-06-30T06:15:27Z

For the past 6 months I’ve been spending most of my time building a new service called Sled. I’ll write more about Sled in the coming weeks, but for now all you need to know is that Sled is a collaborative list making tool for small groups.

Sled is built entirely in JavaScript, both on the back-end and front-end. The back-end is build using Node.js (aka Node), the relatively new server-side JavaScript platform. There is some published experience available about building web applications and services using Node, but the experience available is focused on using Node for real-time applications like instant messaging, chat rooms, and games. It is unusual to hear someone recommends Node for building a new web site if it is not focused on real-time.

I expect this to change very fast.

While Sled has a bit of real-time functionality in the form of live list updates from other participants, the core experience is still closer to a traditional web site. We have two main servers, one serving mostly static pages and scripts for rendering by the browser, and an API server. On the client, we use HTML5, CSS, and lot of JavaScript to talk to the API server and display data.

The next few posts are a random collection of things I’ve learned after 6 intense months of working with Node.

To Node or not to Node

To be honest, I jumped at the opportunity to use JavaScript on the back-end. I’m a C++ guy at heart, coming from a long background of building real-time trading systems on Wall Street. I like C++, and I wish I could still be using it, but last time I tried building a web service in C++, it didn’t work out so well… JavaScript has similar esthetics (no meaningful-white-space for me, thank-you-very-much), it is trivial to read by C++ developers, and is very easy to pick up.

Late 2010, I was introduced to Node by Peter Griess who has been an early participant in the Node effort. At the time, Peter was leading an effort to integrate Node into the Yahoo! Mail back-end environment. Once I figured the overall architecture for Sled, I met with Peter to discuss platform options. I knew Peter was working with Node but was reluctant to use it given its early state.

After an hour of talking, it was clear that I should be giving Node a closer look. I envisioned the API server as a completely statelessness machine with a a simple document-based data store requirements, serving REST calls using as much of HTTP as possible. Node sounded like a perfect fit. For the first time in over a year, I got excited about doing web development on a new platform. Don’t underestimate being excited about your technology.

Ironically, when I asked Peter if I could count on him to help out he told me he just gave notice… Great! But he assured me that the Node community is pretty awesome and that getting help should not be a problem. He was right.

My main reasons for choosing Node 6 months ago were:

JavaScript is a natural choice for developers with C/C++ background
JavaScript-all-around makes code management and reusability easier
Team members can easily move between front-end and back-end
Node is the hottest new platform, making it a good fit for a small project looking to attract talent

Given the early stage of the project, I wasn’t too concerned about performance (which proved to be very high) or stability (which hasn’t been a blocking issue, but see below). I was mostly concerned about libraries and shared experience.

My reasons for choosing Node proved to be right. One exception is code reusability between the back-end and front-end which didn’t really proved to be all that useful. We do a little bit of code reusability, mostly for OAuth 2.0 MAC tokens, but even there, the availability of native cryptographic facilities in Node make the reusable code narrower when moving to the browser.

What I didn’t know 6 months ago, but made me stick with Node:

JavaScript is an extremely productive language, especially for back-end development
The Node community is young which leaves plenty of room to impact core components and library development, and the leads are very open to new ideas and suggestions
It’s fun. Super fun.

Is it Ready for Prime Time?

Last year, when I told people I am going to use Node for sled.com, I was greeted with a lot of suspicion, warnings, and skepticism. Six months later I can say with confidence that it was the best decision I’ve made on the engineering side of the project. It got us off to such a great start and that despite the early age of the platform.

As a developer, Node has been amazingly stable. Our server experienced one week of instability in over 6 months due to a race condition bug in the Node HTTP client layer. This was easy to patch and was fixed within a week. I was very surprised at just how stable the API server have been. I’ve started with version 0.2.6 and now running on 0.4.9, but experienced little to no disruptions during the transition from 0.2 to 0.4. The Node core team has done a great job at keeping the project stable, even in this very early stage.

What isn’t so great is library stability.

There are days when I can’t believe some of these libraries were ever executed, and this extends to some of the more popular modules. This is very much a work in progress and it requires a high degree of direct involvement in the library development. If you are going to use a module, you should join its mailing list and keep an eye on what is going on. Being part of the community is absolutely required if you are going to use Node.

The Community to the Rescue

Node has one of the best communities I had the pleasure of working with over the past 20 years. Everyone is accessible, friendly, and eager to help. It doesn’t matter if you are an expert or a novice, you are going to get attention and the help you need. This is especially true for the major module leads. Getting a library patch within hours is a pretty common thing and it helps if you work with them to pinpoint the problem.

All things considered, whatever instability Node introduces, the community counters with plenty to spare.

More Coming

This will be the first in a series of posts about my on-going experience with Node. I hope people new to Node will find it useful. The next post will focus on Node’s non-blocking nature and how it impacts coding style.

Got an interesting Node project going? I’d love to hear about it.

OAuth 2.0 Redirection URI Validation

2011-06-22T06:25:04Z

Why do we require clients to include the redirection URI when exchanging an authorization code for an access token in OAuth 2.0 (section 4.1.3)?

Consider the following scenario:

Evil user starts the OAuth flow on a legitimate client using the authorization code grant type flow.
Client redirects the evil user to the authorization server, including state information about the evil user account on the client.
Evil user takes the authorization endpoint URI and changes the redirection to its evil site.
Evil user tricks victim user to click on the link and authorize access (using phishing or other social engineering methods).
Victim user thinking this is a valid authorization request (it looks kosher), authorizes access. Access is granted to the right legitimate client. So far nothing is wrong.
Authorization server thinks it is sending victim user back to the client, but since the redirection URI was changed, victim user is sent to the evil site.
Evil user takes the authorization code and gives it back to the client by constructing the original correct redirection URI.
Client exchanges the code for access token, attaching it to the evil user’s account.
Evil user can now access victim user data via his client account.

The way this works, the attacker does not get direct access to protected resources, but it tricks the client into attaching the victim’s access token to the attacker’s account.

Pre-registration of the redirection URI can help a lot, but depends on the matching rules. Since many large providers have open redirectors, the attacker can use those to construct a redirection URI that passes the authorization server validation.

Requiring clients to register their full redirection URI without allowing any variations or partial matching is highly recommended.

Closing a Few More Discovery Loose Ends

2010-11-19T21:26:25Z

Two important specifications reach their final stage last month: XRD 1.0 as an OASIS Standard and Web Linking as an RFC. Both are essential building blocks in web discovery and richer distributed web services. It was a long journey getting both of these published in their respective communities – over 2 years of effort for each.

I would like to thank Mark Nottingham for his leadership and hard work authoring the Web Linking specification. This specification will hopefully help bridge the gap between the various formats (XRD, ATOM, HTML, etc.) and provide a consistent way to type links.

Special thanks go to Will Norris for his editorial lead of XRD 1.0, taking my blog posts and notes and turning them into a useful specification with his own added ideas and improvements, and to Drummond Reed for superbly chairing the XRI TC over the past 5+ years and for supporting my crazy idea to drop XRDS and replace it with something simpler.

(All these Brilliant People at) Facebook Make Me Sad

2010-11-03T07:58:25Z

This is not a post about open, about standards, about privacy, or really any criticism of Facebook in any way. In fact, the problem is just how unbelievable the Facebook team is (in a good way). The sheer strength of their talent is almost unmatched in our industry, past and present. The problem is, all that talent is building something I just don’t care about, and no one is left for anything else.

Facebook doesn’t provide me with anything useful.
When it comes to staying connected to the people I care about, they either live with me, I talk to them on the phone weekly, or have an annual dinner when I visit Israel or New York. This is just enough for me. There is a reason why I am not in touch with people from high school, the army, or film school. We all moved on, became different people, changed context, and lost the common thread that united us at the time. My personal Facebook experience of finding long lost friends is mostly a short awkward exchange followed by a one sided stream of useless information.

When it comes to content, I much rather rely on the editorial board of the New York Times for my news, than what my “friends” find interesting. The idea that people I care about are in any way relevant to my news consumption has never produced useful results. When it comes to news, I want to be “friend” with the editors of the New York Times, and when it comes to buying a digital camera, the reviewers at DPReview.

My family and friends are uniformly challenged when it comes to world affairs or digital cameras. Maybe it is just me, but in my offline world, information and sharing works perfectly fine without Facebook. As for casual chatting, games, and leaving comments on people’s photos – I choose to spend my free time doing something else. Instead of dealing with people’s shit, I rather handle chicken shit.

I am in no way suggesting that almost 600 million people are wrong. The massive and highly engaged Facebook user base clearly gets value and satisfaction from the product. I am also in no way critical or judgmental about those who find value there. Good for them! But for me, there are so many other unsolved problems in the world, and they have little to do with social.

Last month at Open Web Foo, sitting outside with about a dozen of some really smart and highly connected folks, I asked people to name three “wow” experiences they had with new web products in the last year. Most had none to share. I asked them to think back 3 years and the list filled up in minutes.

Forget about solving the world’s problem – if you don’t care for Facebook, the web it just boring. It’s stale.

There are many reasons why engineers want to work for Facebook, from the potential windfall to learning just how they are able to ship so much technology so fast. It is an engineering dreamland. But there is one great reason why they shouldn’t: because Facebook will be great without them, but the web might not.

The Growing Web Identity Crisis, Courtesy of Facebook

2010-11-02T21:33:05Z

A concerning trend is showing up in recent TV and print advertisements of companies using their Facebook profile pages as their web identity instead of their own domains. Most of these companies are big corporations with a well-established web presence. Using social networks to connect with consumers and promote brands is not new, but using these identities as the primary corporate web identity is new.

For the past few years, the web identity community has been focused on the engineering side of a distributed web identity platform, developing protocols and tools to improve its usability. These efforts have mostly failed due to lack of clear consumer demand, low value, and an unusable user experience. At the same time, consumers have been eagerly embracing proprietary and closed solutions such as Facebook Connect and Sign-in with Twitter.

While engineers love to debate protocols and solve engineering problems, the core issue has always been the identifier. What should people use to identify themselves to other users and when logging into web services. Such identifier solutions included HTTP URLs (such as your blog), email addresses, proprietary screen names, social networking accounts, PKI certificates, or new naming schemes (there is always a new identity startup just around the corner looking to sell you new moon lots).

The recent trend in corporate web identity is to move away from their own web presence to a page on Facebook or Twitter. This is likely to make deploying a distributed web identity solution even harder. Web users have been voting with their clicks and opted to use their social networking profiles as their web identifiers. This is largely due to a clean, simple, and useful design offered by Facebook and Twitter. But the growing participation of corporations in this space is signaling that consumers are now perceiving these closed platforms as a natural part of the “open” web infrastructure.

What was initially unacceptable – to put another company name on your ad – is now common practice: ‘facebook.com/target’ instead of ‘target.com’. Target no longer fear subordinating their identity to that of Facebook, because for most web users, the ‘facebook.com/’ part is just the new form of ‘http://’ – something they don’t care or understand which comes before the stuff that actually matters. Facebook in this context, is completely transparent to most users.

Facebook has always claimed to be a platform and a utility, and this is the most obvious manifestation of this goal. The real threat to the open web is not from proprietary protocols and closed networks, but from ‘facebook.com/’ and the ‘@’ screen name prefix becoming another well-known prefix like ‘http://’ and ‘www.’.

OAuth Bearer Tokens are a Terrible Idea

2010-09-29T19:09:29Z

My last post about the lack of signature support in OAuth 2.0 stirred some very good discussions and showed wide support for including a signature mechanism in OAuth 2.0. The discussions on the working group focused on the best way to include signature support, and a bit on the different approached to signatures (more on that in another post).

However, the discussion failed to highlight the fundamental problem with supporting bearer tokens at all. In my previous post I suggested that bearer tokens over HTTPS are fine for now. I’d like to take that back and explain why OAuth bearer tokens are a really bad idea.

But first, some live entertainment:

Facebook developer ‘wesbos’ writes:

Hey guys, I’m using the freshly downloaded PHP API

I’ve got eveything setup and when I login to authenticate, I get this error:

Fatal error: Uncaught CurlException: 60: SSL certificate problem, verify that the CA cert is OK. Details: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed thrown in /Users/wesbos/Dropbox/OrangeRhinoMedia/reporting/src/facebook.php on line 512

To which ‘Telemako’ replies (emphasis is mine):

[…] try this one:

$opts[CURLOPT_SSL_VERIFYPEER] = false;
$opts[CURLOPT_SSL_VERIFYHOST] = 2;

And ‘philfreo’ asks:

Is there any good reason why these shouldn’t be a part of the core default $CURL_OPTS?

Well, philfreo, Telemako, and wesbos, you have just turned off one of the most important protections SSL provides: defense against MITM attacks.

Putting All Your Eggs in One Basket

The main problem with using OAuth bearer tokens is the fact that they rely solely on SSL/TLS for its security. Bearer tokens have no internal protections. Just like cash, whoever holds the token is its rightful owner, and proving otherwise is impractical. OAuth 1.0 signatures evolved from the basic requirement not to mandate SSL/TLS. But using signatures goes beyond just a deployment consideration.

In his blog post “defending against your own stupidity”, Ben Adida (a highly respected security expert) writes:

Consider the utility of a safety parachute. A determined attacker trying to kill you will obviously sabotage the safety parachute just as easily as he can sabotage the primary one. So, does that mean you might as well jump without a safety parachute? Of course not. You want to take into account not just the worst-case attacker, you want to take into account your own stupidity. A safety parachute means that, if you packed your primary wrong, you can still live. Defense in depth, as it’s more commonly known in the security community, is usually not about building the 12 layers of security around the “Die Hard” vault that a skilled attacker has to vanquish, one by one. Defense in depth is the humble realization that, of all the security measures you implement, a few will fail because of your own stupidity. It’s good to have a few backups, just in case.

No matter what you do, if for any reason the SSL/TLS layer is broken, the entire security architecture of OAuth bearer tokens falls apart. In a world moving steadily towards multiple means of authentication and verification, this is a step backwards.

SSL/TLS is Harder than You Think

It is common knowledge among security experts that implementing SSL/TLS is notoriously hard. This is why very few attempt to write their own libraries. However, equally well established is that many clients implement SSL/TLS in a way that voids its most valuable defense: that against a man-in-the-middle. If anyone can get between your client and the server, they can capture the token and use it at will.

The example above shows one of two common ways in which SSL/TLS security is compromised. I would love mock the developers quoted but it is a mistake I have made myself three years ago when writing my own OpenID library. I had no problem getting the signatures done, but had a hard time with bad certificate errors. Some were due to local configuration (getting root certificates configured is still a hard thing for most developers to do), but also due to poorly configured servers. My solution, just like the suggestion above, was to ignore these errors.

The problem with SSL/TLS, is that when you fail to verify the certificate on the client side, the connection still works. Any time ignoring an error leads to success, developers are going to do just that. The server has no way of enforcing certificate verification, and even if it could, an attacker will surely not.

This is not new. Ben Adida described this exact issue when reviewing the original WRAP proposal:

But wait, you say, don’t worry, the token is sent over SSL, so it’s all good.

Right. What’s going to happen when someone “forgets to turn on SSL”, which is all too common when security is abstracted out “somewhere down in the stack.” Or when we stop dealing with those pesky certificate errors and just choose not to validate the cert, which will leave the protocol wide open to network attackers who can now literally play man-in-the-middle just by spoofing DNS on a wifi network, capturing the token, and replaying it to access all sorts of additional resources, effectively stealing the user’s credentials.

This might actually be worse than passwords, because at least you can work to educate users about SSL (and after their Facebook account gets hacked, they might actually care), but it’s very hard for users to gauge whether web applications are doing the right thing with respect to SSL certs when the SSL calls are all made by the backend which has trouble surfacing certificate errors.

The second common potential problem are typos. Would you consider it a proper design when omitting one character (the ‘s’ in ‘https’) voids the entire security of the token? Or perhaps sending the request (over a valid and verified SSL/TLS connection) to the wrong destination (say ‘http://gacebook.com’?). Remember, being able to use OAuth bearer tokens from the command line was clearly a use case bearer tokens advocates promoted.

Who is in Charge of Your Service Security?

The main difference between using bearer tokens and signatures is in who they put in charge of enforcing the service security. Signatures ensure that mistakes don’t compromise the token because the secret is never sent with the request. At most, an attacker can use the captured request once, and for enhanced security, signatures can and should be used together with SSL/TLS for a multi-layer protection.

Yes, developers do stupid things, like posting their client credentials (API keys) on public lists and in open source code. But I have yet to see developers expose token secrets like that. It is really hard to expose OAuth 1.0a token secrets by mistake (unless that mistake is leaving your entire database open for public review, in which case, nothing will help you).

Bearer Tokens are not Just Like Cookies

The winning argument in favor of using bearer tokens has always been cookies. Bearer tokens have the same security properties of cookie authentication, as both use plaintext strings without secrets or signatures. However, there is one big difference – the client developer.

For many years, browsers made it insanely easy to ignore bad certificates. It took a lot of time and many attacks to make it hard and scary to approve bad certificate exceptions. Client developers are usually not at the same level as browser security developers. While a JavaScript OAuth client enjoys the same quality code that the browser uses to verify certificates, other clients do not.

The Ironic Solution

We know developers don’t read security considerations sections. That’s a fact of life. Developer only read the parts they need to implement and ignore the parts that ask them to think. Only large companies with dedicated security teams pay much attention to that, and they usually do their own analysis. So warning developers to check the certificate is not likely to accomplish much.

What’s the alternative? SDKs!

If your developers cannot be trusted to implement your API correctly, it is common to offer them a library instead. I hope you can see the irony here. If you solve the SSL/TLS deployment issue with an SDK, why not just implement signatures? One argument used against signatures was that even with libraries, people will try to write their own code. Well, that’s a scary proposition.

Security is Hard

Ben Laurie, one of Google’s top security experts and the OpenSSL project lead writes in his explanation why WRAP “is just so obviously a bad idea, it’s difficult to know where to start”:

The idea that security protocols should be so simple than anyone can implement them is attractive, but as we’ve seen, wrong. But does the difficulty of implementing them mean they can’t be used? Of course not […] every crypto algorithm under the sun is hard to implement, but there’s no shortage of libraries for them, either.

As I have pointed out before, the ‘signatures are hard’ argument dies with Twitter’s recent deployment of OAuth signatures as a requirement.

But Everyone Else is Doing it

My favorite argument in favor of bearer tokens, and the most ridiculous one, is that Facebook, Google, Microsoft, and Yahoo! are all doing it, so that must mean its good. After all, they must have reviewed it and decided it was safe.

The other variation of this argument is that bearer tokens are already widely deployed so we need to support them. All I have to say is quote the obvious: “if your friend jumped off a roof, would you?”

Facebook is using MD5 in their current API, probably because that is what Flickr used when Facebook copied their API authentication scheme. See the pattern? New services simply follow what other established companies are already doing, without questioning the reasoning behind it. Big companies have resources to make these decisions, but they rarely apply as-is to others, and it is simply irresponsible to ignore their leadership position.

This is how we got stuck with Basic authentication and cookie logins. This is what RFC 2617 has to say about Basic HTTP authentication:

The Basic authentication scheme is not a secure method of user authentication, nor does it in any way protect the entity, which is transmitted in cleartext across the physical network used as the carrier.

And yet, it is still one of the most widely deployed authentication scheme on the web. People don’t read specifications, they just follow the market.

Actually, Not Everyone is Doing it

OAuth 1.0 had bearer token support alongside signatures for three years now, and yet, it is barely used. Twitter could have deployed OAuth 1.0 as specified in RFC 5849 section 3.4.4 but chose not to. The claim that bearer tokens are a new feature is false. All OAuth 2.0 does is clean it up and present it in a more accessible way.

The problem is both with the specification and the market hype. By positioning bearer tokens as the default choice for 2.0 developers, and by highlighting the big vendors deploying it, developers are less likely to think for themselves, especially given the allergic reaction many developers have to signatures.

I still have hope that early and future adopters of OAuth 2.0 will make the right decision and not deploy bearer token support for their core APIs. I have never suggested to outright ban bearer tokens. They provide a useful way to experiment with an API, to try out ideas, to test code, and to make API calls that have little to no security requirements. For example, it is perfectly fine to use bearer tokens for any Twitter API call that returns user-specific data that is publically available (like your followers list). Not so fine to use it to update your status or read private messages.

Now What?

I have concluded that the OAuth working group is not a productive place to have this debate. People are much too entrenched in their positions and with existing live deployments, a committee is not the way to reach consensus. Instead, I believe that an open public debate and market pressure is the best way to go. This is where I hope others will join me in asking their service providers to take their security more seriously.

With regard to the specification, I have proposed (and so far received wide support) to make an editorial change that will separate the ‘how to get a token’ part from the ‘how to use a token’ part. This is not a new idea. By doing so, we will enhance the modularity of the protocol and remove any bias for or against bearer tokens and signatures from the specification. Instead, developers will be presented with two or more alternatives, and will have to make their own decision about what is right for their platform. This proposal is still pending working group consensus.

I am sure this will not be the last word on the subject.

More OAuth Nonsense

2010-09-23T06:10:49Z

ComputerWorld is the latest to run a scary story about OAuth 2.0 and how insecure it is. Unfortunately, instead of doing their homework and paying attention to my post, they borrowed a bunch of my quotes (almost half the article), added some original nonsense, sprinkled a few errors, and gave it a sensational headline: “OAuth 2.0 security used by Facebook, others called weak”.

OAuth 2.0 without signatures works just fine for companies like Facebook, because their developers hard-code their API endpoints. There is no danger what-so-ever for an access token to leak or get phished (any more than if they used signatures) because Facebook uses HTTPS for their OAuth 2.0 API. This is why I titled one of the subsections: “Why None of this Matters Today”. My post was about discovery and long term security improvements of the web – not that there is anything broken about today’s implementations.

Let me make this crystal clear: OAuth 2.0 without signatures over HTTPS is perfectly secure for proprietary APIs where the client only sends the access token to a hardcoded endpoint, and uses HTTPS properly.

My favorite part is the quote from Google’s Eric Sachs (who, BTW, is not a co-author of any OAuth specification), explaining why having no signatures is a good thing:

These guys are not experts in security. They can have a hard time understanding security models.

This is exactly the problem. OAuth 2.0 moves some of the enforcement requirements (again, when used with dynamic service discovery) to the client, and the client developer is rarely a security expert. This is why the specification has to assume that the client developer is going to make mistakes and do everything possible to render them harmless.

Raise your hand if you ever wrote a client application using HTTPS and got tired of exceptions about bad certificates, so you just ignored them silently. Unless your HTTPS code is written in a way that completely blocks the connection if there is any error, it might not do you much good. HTTPS is great when done right, and this is just one example of the difficulty in trusting clients to do the right thing.

To be clear, I don’t want mandatory signatures in OAuth 2.0. I think there are valuable use cases for the current bearer token approach. And I agree that developer ease is an important design factor. Signatures were not mandatory in OAuth 1.0a either, but they are an equal part of the protocol, as they should be.

Given the recent support for putting signatures back in (as an optional component), I have pushed back the publication of draft 11 until we can add a signature proposal to the core specification. I expect it sometimes in late October.

Goodbye Open (and Why I’m Staying at Yahoo!)

2010-09-17T22:43:22Z

It’s that time again, to move on. The past three years have been a roller-coaster. Coming from a small startup after a decade in financial services technology, I got to learn, contribute, lead, and provoke open web development. My standards participation landed me a great job, relocated my family to the West coast, and introduced me to a lot of amazing people. It has been awesome.

Over the past couple of months I have been steadily phasing out my open specifications and standards involvement. The OAuth 2.0 core specification is the only thing I am still working on (OAuth is a keeper). Everything else has either fizzled away or lost its interest to me. This should not come as a surprise to anyone who talked to me or read my posts over the past few months.

First, let’s get the most important detail out of the way. I will be working on a new “startup”, building from ground up a new consumer web experience. No, I am not leaving Yahoo! In fact, I’m planning on sticking around for a while. This wasn’t an obvious conclusion, but one that I am extremely excited about (I’ll explain why in a second).

A startup at Yahoo! you ask? Well, you will have to wait a bit longer for more on that. We’re still working out the details. For now, I just want you to know that I am currently hiring a front end developer (in the Bay area). The project is probably going to use Node.JS, CouchDB, HTML5, JavaScript, OAuth, and other cool new stuff. This is a chance to combine startup environment with job security and stability. I plan to write a lot more about working at Yahoo! and this new project soon.

Taking Inventory

In the tradition of recoding my thoughts as I set on a new adventure, these are some of the things I worked on over the past couple of years (and I do mean to brag, a little):

OAuth – Leading author and editor of the 1.0, 1.0a (RFC 5849), and 2.0 specifications, as well as many articles and tutorials. Shepherded the transition from an open community to the IETF. Negotiated licensing terms. Coordinated the handling of a critical security flaw.
Open Web Foundation – First elected president, co-founder, and past chair of the Legal Affairs and Elections committees. Established bylaws, and managed the foundation corporate registration.
XRD – Lead architect and co-editor of the XRD 1.0 OASIS XRI-TC standard (pending). Previously authored the XRDS-Simple specification.
Well-Known URIs – Co-authored RFC 5785 for establishing a URI prefix for well-known locations, and a IANA registry (serving as the Designated Expert approving new requests).
Host-meta & LRDD – Authored a proposed RFC for a method locating host metadata for Web-based protocols, including a unified link-based resource descriptor discovery method. Work coordinated across many communities and standards bodies including W3C TAG.
OpenID – Leading participant in the redesign of the discovery layer and identity services.
Metalink – Served as document shepherded in the IETF for RFC 5854.
WebFinger – Co-author of a simple web-identity layer using account URIs (email-like identifiers) for the discovery of social and personal web identity information.
Portable Contacts – Helped with the initial draft. Facilitated discussions with the IETF vCard community.
OpenSocial – Represented Yahoo! in the foundation creation process. Provided the initial design for the IPR framework and governance model.
Discovery – Participated in specification efforts including Salmon, Activity Streams, OExchange, DiSO, Web Linking, and others, with focus on discovery services and architecture.

Is Open Dead?

Absolutely not. But I think the kind of open I was working on is mostly gone. This is not a bad thing, just the result of the changing industry, people’s careers, and economic conditions. For the most part, the movement that started with OpenID and OAuth is largely over. All the cool kids got grownup jobs and have been mostly missing. Think of the people you used to hear from on a weekly basis, and then try to remember when was the last time they had something new or provocative to say.

I have written extensively about the changing industry and the role of Google and Facebook in the new environment. Standards development goes through cycles of products and explorations. During the exploration phase, people come together with ideas and try to create a new marketplace by working together to invent new capabilities and opportunities. Email, HTTP, and HTML are some well-known examples. During the product phase, companies with specific existing requirements come together to either unseat the market leader, or to solve specific problems where there is no competitive advantage to go at it alone.

Three years ago the social space was a happening scene. Every day a new startup emerged with a new idea on how to make the web more social. Companies focused on specific communities or activities, and there was plenty of excitement. As the space matured, we now have Facebook as the clear winner with a lead that will be impossible to beat anytime soon. Facebook is to social networking what Google is to keyword search. Both are too far ahead of everyone else in their respective areas.

Got Product?

I am sure in a year or two we will go back to the exploration phase, but for now, we are deep in the product phase, and if you don’t have a product, you don’t really belong there. I have not been a fan of Yahoo!’s social strategy for most of the past two years. The vision coming from the top these days is a lot more promising, exciting, and aligned with where I think the company should go, but this was not always the case.

The reality is, that when it comes to the kind of technologies I’ve been involved in, Yahoo! has not been an important player. This did not impact my ability to lead during the exploration phase, but now without a leading product, it is very hard to be effective. If you want to develop new social web standards, you absolutely have to work at Facebook or Twitter. That’s where all the action is. Google might be trying its best, but in practice no one cares about social technology coming out of Google. They have also lost their leadership in most of the groups I have been active in (and not due to lack of trying).

There is still interesting work done by Blaine Cook, Status.net, and the federated social web folks, but it’s all too experimental, and I don’t see it anywhere near mass market anytime soon.

Standards is a Horrible Career Path

Late last year I came to the conclusion that my ability to be effective at my job is coming to an end. Yahoo! has been consistently supportive of my work, provided me with a wealth of resources, and gave me absolute freedom to “make the web better” (that’s how one of my managers described my job). This is very unusual, and something Yahoo! deserves a lot of credit for. Yahoo! repeatedly stepped up to support the community and provided resources without asking what’s in it for them, because it was what was best for the web. This is very rare in our industry.

I was faced with two options: find a new job where I can be effective again, or move on. Given my passion (and success) at developing open specifications and communities, I spent a considerable time and effort looking for a similar opportunity elsewhere. I talked to all the usual suspects, got some offers, got ignored a lot, and at the end arrived at the conclusion that it’s time to move on from standards. I do want to say that of all the companies I talked to, I was impressed with how respectfully Facebook handled the interaction. Can’t say that about some of the others.

I also realized just how destructive choosing standards as a career path can be. The standards world is very demanding and will suck every free minute you have. Most people contribute very little, and at the end, a handful of people end up carrying all the load. The problem is, no one wants to foot the bill for those suckers. Only a handful of very large corporations (mostly telecommunication and hardware) support employees doing standards full time, and mostly to serve their self-interest.

Anyone who ever met me knows that I am not the right person to represent a company line. I have my own voice, it’s loud (sometimes obnoxious), and I’m not shy of using it. I don’t know of any other company that allows its employees to speak their mind so freely and openly, even when they are in disagreement with the company’s official policy. Even our legal team has policies in place to enable that. I have often been out of alignment over Yahoo!’s OAuth and OpenID strategies. But at no time was I told to shut up, something at least one of my collaborators at Google cannot say.

At the end, especially during hard economic times, finding a role focused on standards is hard. It is especially hard when you have views that are not likely to always be in line with those of your employer. Kids – stay away from standards as a career.

It’s All About the Product

I miss building stuff. I miss writing code and seeing how something starts from nothing and progress into a working system. With the exception of the last three years, I have been building applications since I was 8 years old, using my first Sinclair ZX81 computer. When I was 11 a friend and I wrote a palm reading program. We got into trouble when we set up a stand at a local fair and gave someone a printout stating that her intelligent was below average. Hey, not my fault her head-line was so short.

More than OAuth, my passion the past few years has been discovery. It is very rewarding to see some of my work finally making its way to actual products. But most of the time it was just frustrating, putting all this effort in without any actual commitment from those building products. I still think discovery is going to be more and more critical as the web grows and becomes more distributed, but it will take a while longer before my vision can be fully appreciated.

Why Yahoo!?

That’s a great question, and one that I have been debating for almost a year now. It was a hard decision, but once I reached it, it was clearly the right decision, and I am super excited about the coming year. To get to my conclusion, there are a few factors you need understand. When considering a new employer, I mostly care about: their products, their culture, my manager, my team, and how they balance work-life. Technology, compensation, location, reputation, and IPO prospects have never made a difference to me.

So let’s go through them one by one.

Products –

Yahoo! is still working to define what it is. I have heard the official description but like most people, am still trying to figure out what it really means. What I do know is that we have the biggest audience on line, a long list of best in class properties, a leading brand name with unparalleled trust (no one is afraid of Yahoo! trying to take over the world), and a user profile that is exactly the people I want to build products for – normal everyday web users. If you get the core Yahoo! user base to use a new experience, your impact is massive – far beyond just web habits.

Facebook, Google, and Twitter are all one hit wonders. Granted, these are some unbelievable wonders, and I have nothing but awe and respect for what these three companies are doing in their space. But let’s face it, they are known for one thing (Google has a few new promising areas like Chrome and Android), and working there most likely means focusing on improving an existing product rather than building something new.

I rarely use Twitter or Facebook. They don’t add much value to me. That’s in no way criticism of their products, just that I care a lot more about what people say on BackyardChickens.com than on Twitter or Facebook. Same goes for the people in my family. One of the reasons Facebook is such a success is the fact their employees are truly passionate about the product and use it intensely, and I think that that alignment is important.

Working for Yahoo!, especially these days, offers more opportunities to work on something new and exciting than anywhere else. It offers unmatched access to resources and audiences, and a management team that is eager to hear new ideas from every level of the organization.

Culture –

If it wasn’t clear by now, I value openness, directness, flat hierarchy, equality, diversity, and individuality. This list alone rules out a bunch of companies. Google’s approach is to build an army of (really smart, talented, and somewhat socially inadequate) clones. When you are hired, you don’t know what you are going to work on. Google’s hiring process is designed to make sure they hire great engineers that can do anything. This allows them to shift resources around as needed quickly, but in the three times I engaged Google over a job, I could not get past that. I will not take a job where I am just another engineer.

Facebook is much better, but their emphasis is mostly on coding skills alone. I could not figure out what Twitter’s culture is like (and I’m not sure if they know either).

Yahoo! scores high on every one of my cultural values. The problem at Yahoo! is that the culture has been taken hostage by complaining, ranting, sarcasm, irony, and a generally depressing mood. The underlying culture is very much there, but it is hard to enjoy it given the general atmosphere. Part of the problem, is the constant hammering of the tech press, making a big deal out of every departure (much more than at other companies).

This is where Yahoo!’s collegial culture is counterproductive to its own best interest. Consider some of the recent high profile departures. Yes these are all smart, talented individuals. But with a few exceptions, these are mostly people who got pass on in a recent reorganization, failed to deliver a product, did not fit in a company focused on mass market consumer experiences, were offended that someone else got a better role or more control, or been there for more than 5 years. In other words, this is corporate business as usual.

The problem is, because Yahoo!s are so nice (people often wonder why they keep me), they will never leak out and share these negative realities to balance the news (we are known to leak plenty of other stuff). This is great for the individuals working there, because at some point, we all move on, but at the moment, it is taking a toll.

I am in no way blaming the press for any of the underlying issues, or for doing their job highlighting them. I don’t think the coverage is fully balanced (“People are running away from Yahoo!” vs. “Facebook stole another Google engineer”), and it is clearly one of the main factors casing the current mood. I only mentioned this as an example of how the culture and mood are linked.

My solution is simple. Two months ago I decided to try and stop complaining. No more sitting at URL’s (our café) bitching about everything we suck at. For now, it seems to work.

My Manager –

I think most people will consider this entire post as one big ass kissing. I hope those who know me see this for what it is – a public venting of my views, frustration, admiration, and random rants. I like to share my views, even when others find them inappropriate. It has made me plenty of enemies, but no one has ever claimed not to know where I stand or where they stand with me.

Keep in mind that I’m the guy who, on his first week at Yahoo!, was featured on Valley Wag under the headline: “New Yahoo: Joining up without severance package would be like ‘running into a burning building’” (I am still the number one search result for ‘running into a burning building’). With that said, I am going to simply state that my manager, Mike Curtis (head of Mail engineering), is awesome. I recently had the rare opportunity to pick my own manager and I think that says it all. I am also happy about the chain of command between me and the CEO.

My Team –

My discussions around a new role outside Yahoo! did not reach the team evaluation stage (with one exception at Facebook, where the team was great). So while important, it was not a big factor. What was, is my ability to build my own team, which is something I have always enjoyed doing. At Citibank, I twice built large teams building high frequency trading systems, and it is something I miss doing. I am not going to be building anything this big anytime soon, but I get to hire a couple of people today and that’s plenty fun.

Work / Life Balance –

This is Yahoo!’s number one asset and number one liability. If you are an employee, you will not find a better place to have a family, to have a hobby, or in other words, to have life beyond work. Between company events, benefits, and overall atmosphere, Yahoo! is the most welcoming place I have ever worked (or interviewed with) when it comes to having a family (our twins are turning 6 next month).

Beyond its culture, the reason for that is the profile of the employees. Being one of the few “old” web companies around, and currently attracting less young blood than its competitors, Yahoo!’s employees tend to have a family and are not the kind of people usually joining a startup. The problem, of course, is that this makes it harder to move fast and innovate at a competitive rate. Every time I visit Facebook I feel like (at 36) I’m the oldest guy in the room – the place is like a college campus.

But with the new management’s determination to innovate, and to give anything a try to accomplish that, I really believe Yahoo! is in a unique position to attract a large group of brilliant engineers, those not interested in the bootstrapping startup life, but innovative and passionate about building the next generation of web experiences. Being the place where you don’t have to choose between spending time with your family and changing the world through web innovation, is something Yahoo! has a unique opportunity to become, more than anyone else. We are clearly not there yet, but this is something I want to work on.

This is also why I believe Yahoo! has a better chance of understanding our audience, the same way Facebook truly understand theirs. Our company profile is very much the audience we seek to serve – everyone.

On to the Next Great Thing

It took me a while to get here, and after a yearlong search, I’ve decided to stick around and it feels right. There are still more things to figure out about my new project and the company, and like any “startup”, anything can happen and this can be short lived or a huge success (or anything in between). Many people I talked to these last few days told me I’m nuts, day dreaming, or naïve. Maybe I am.

But I’m having more fun than everyone I know, I got a great job, I’m paid well, my employer truly cares about my well-being, I get to spend quality time with my family, and I even get to play farmer in my spare time (chickens, ducks, bees, large garden and an orchard).

Can you say that, without qualifications, about your job?

I’m hiring a front end developer (you can start tomorrow), and Yahoo! has plenty of great jobs available in both new and core products. If you are interested, and available in the Bay area, email me at eran@hueniverse.com.

This blog will continue to highlight open standards, because that’s something I am still very much passionate about, but it will also offer more insight into life at Yahoo!, my new project, and whatever else is on my mind. This is going to be yet another fun ride. Stick around.

OAuth 2.0 (without Signatures) is Bad for the Web

2010-09-16T06:07:00Z

OAuth 2.0 drops signatures and cryptography in favor of bearer tokens, similar to how cookies work. As the OAuth 2.0 editor, I’m associated with the OAuth 2.0 protocol more than most, and the assumption is that I agree with the decisions and directions the protocol is taking. While being an editor gives you a certain degree of temporary control over the specification, at the end decisions are made by the group as a whole (as they should).

And as a whole, the OAuth community has made a big mistake about the future direction of the protocol. A mistake that is going to make OAuth 2.0 a much less significant agent of change on the web.

All this Crypto Business

OAuth 1.0 allows application developers to sign requests. Signatures remove the need to send plain-text secrets on insecure (or secure) channels. Instead of sending the secret with the request for the other side to compare against their copy of the secret (similar to how passwords work), the secret is used to calculate a value which cannot be converted back the secret itself. Instead, the signature can be verified by someone with a copy of the secret.

When performing an irreversible calculation that the other side can verify, signatures protect secrets by simply never sending them on the wire. By removing the need to send secrets, applications don’t need to rely on other protocols (such as SSL/TLS) to protect the plain-text secrets. That’s not all signatures provide, but more on that later.

To sign a request, developers have to follow a list of steps in a very specific order and with much care (which often feels like battling a dragon). The smallest mistake causes the entire request to fail. While the OAuth 1.0 signature process could have been somewhat simpler (no double encoding, different sorting, no URI parsing into query parameters, etc.), any time developers need to canonicalize data, stuff breaks. Even beyond the complex math, cryptography is hard because it is generally unforgiving. It does not tolerate mistakes.

WRAP and the Stupidity Threshold

After deploying OAuth 1.0, many companies discovered the cost of supporting OAuth 1.0 due to mismatching signatures. OAuth 1.0 looks simple enough for developer to code from scratch instead of using a library (as opposed to SSL or TLS which no one in their right mind will try to write from scratch). When developers write their own code, they are likely to get one small detail wrong. It didn’t help that the specification was vague and implicit about many important details (which since has been corrected in the RFC).

Ironically, the bigger the company was, the more resources it had, and the least interesting and useful API it offered, the louder the complaining about OAuth signatures was. This had an easy and straightforward solution: provide better libraries to your developers as well as better (or any) debugging tools. Alternatively, make your API so valuable that developers will be motivated to struggle through it and figure it out. Unfortunately, this was not the solution the people behind WRAP had in mind.

At the heart of the WRAP architecture is the requirement to remove any cryptography from the client side. The WRAP authors observed how developers struggled with OAuth 1.0 signatures and their conclusion was that the solution is to drop signatures completely. Instead, they decided to rely on a proven and widely available technology: HTTPS (or more accurately, SSL/TLS). Why bother with signatures if instead the developer can add a single character to their request (turning it from http:// to https://) and protect the secret from an eavesdropper.

Much of the criticism that followed focused on the fact that WRAP does not actually require HTTPS. It simply makes it an option. This use of tokens without a secret or other verification mechanism is called a bearer token. Whoever holds the token gains access. If you are an attacker, you just need to get hold of this simple string and you are good to go. No signatures, calculations, reverse engineering, or other such efforts required.

As Secure As a Cookie

WRAP was based on a simple, and powerful argument: bearer tokens are already a core web architecture. While far from ideal, the WRAP security model was directly based on cookies – the authentication layer behind almost every web application. Why bother to create something more secure if it makes it harder for developers to use, while not actually improving the overall security of the service. As long as a site offers both an OAuth API and a human web interface (i.e. a web site), the overall service will only be as secure as its weakest part – the cookie-based authentication system.

The problem with this argument is not today, but 5 years from now. When trying to propose a new cookie protocol, developers will make the same argument, only this time pointing the finger at OAuth 2.0 as the weakest link. Removing signatures and relying solely on a secure channel solves the immediate problem, and maintain the same existing level of security. But it lacks any kind of forward looking responsibility, and the drive to make the web more secure. It’s a copout.

What makes this more frustrating is that the people behind WARP are some of the brightest security minds on the web. These guys know exactly what they are doing, and it’s not like they don’t care. They just gave up and decided that the best they can do is maintain the status quo. They are also representing a large and powerful coalition of big companies too lazy to work a little harder by helping their developers use signatures successfully.

Doesn’t HTTPS Solve Everything?

HTTPS guarantees an end-to-end secure connection. The implementation and deployment details are critical to ensure that, but when done correctly (which is not always the case), is a great solution. What HTTPS provides is a secure channel. Any secret, password, or bearer token sent over HTTPS is protected and cannot be compromised by an attacker listening in on the line. HTTPS allows a client to send a secret to its desired destination securely.

However, HTTPS can’t help if the client’s desired destination is a bad place. HTTPS doesn’t help prevent phishing attacks because anyone can get an SSL certificate and show the secure icon in the browser. The fact you are using a secure channel doesn’t mean the entity on the other side is good. It just means that no one else can listen in on it (just the bad guys). If a client sends their bearer token to the wrong place, even over HTTPS, it’s game over.

Another issue is that the OAuth working group could not even reach consensus on actually requiring HTTPS, leaving it as an recommendation for services to decide. Even OAuth 1.0 requires HTTPS for its plain-text flavor, which was added to get it published as an RFC. Instead, OAuth 2.0 is satisfied with just a warning. OAuth 2.0’s solution is to allow (but not require) access tokens to be short lived. By limiting the bearer token’s lifetime, stolen tokens are only useful for a short period of time, limiting the potential damage.

Why None of this Matters Today

OAuth today is used together with proprietary web service APIs. There is little to no interoperability across these services (Facebook API used only on Facebook, etc.) and almost no clients performing discovery of any kind. Because the API endpoints are hard coded into the client, when combined with HTTPS, there is no risk of leaking the tokens. In this setup, the client does not need to do much thinking about where to send tokens and how to protect them.

Unlike cookies which are sent to the server based on a somewhat complex set of client rules, OAuth clients today don’t use any rules. Instead they use a single token for an entire service with all API endpoint preconfigured. There are no new subdomains to handle, or really any kind of unexpected or dynamic interaction. In this environment, bearer tokens over HTTPS are just fine.

Why All of this will Matter Soon

As soon as we try to introduce discovery or interoperable APIs across services, OAuth 2.0 fails. Because it lacks cryptographic protection of the tokens (there are no token secrets), the client has to figure out where it is safe to send tokens. OAuth reliance on the cookie model requires the same solution – making the client apply the security policy and figure out which servers to share its tokens with. The resource servers, of course, can ask for tokens issued by any authorization server.

For example, a protected resource can claim that it requires an OAuth access token issued by Google when in fact, it has nothing to do with Google (ever though it might be a Google subdomain). The client will have to figure out if the server is authorized to see its Google access token. Cookies have rules regarding which cookie is shared with which server. But because these rules are enforced by the client, there is a long history of security failures due to incorrect sharing of cookies. The same applies to OAuth 2.0.

Any solution based on client side enforcement of a security policy is broken and will fail. OAuth 1.0 solves this by supporting signatures. If a client sends a request to the wrong server, nothing bad happens because the evil server has no way of using that misguided request to do anything else. If a client sends an OAuth 2.0 request to the wrong server (found via discovery), that server can now access the user’s resources freely as long as the token is valid.

It is clear that once discovery is used, clients will be manipulated to send their tokens to the wrong place, just like people are phished. Any solution based solely on a policy enforced by the client is doomed.

No Discovery for You

Without signatures, OAuth 2.0 cannot safely support discovery. It is a waste of time and a risky business. Clearly, the OAuth community today does not care enough about discovery and interoperable services to do something about it. The cryptographic solutions proposed so far are focused on self-encoded tokens and other distributed systems, based on narrow use cases promoted by the likes of Google, Microsoft, and a few other enterprise-focused companies.

Without discovery, smaller companies will have a harder time getting their services accessible (e.g. when importing your address book from any provider, not just the big four).

Now What?

I am not advocating throwing OAuth 2.0 out, starting over, or requiring signatures. All I have ever advocated for is the inclusion of a basic signature option in the core specification, in the spirit of OAuth 1.0. The 1.0 signature isn’t perfect, but as the Twitter developer community demonstrated, is clearly within reach.

Twitter a Hot Princess, Google an Empty Castle

2010-09-16T06:06:05Z

Over the past two years I have been arguing that the problem with supporting OAuth 1.0 signatures wasn’t with the signatures, but with the lack of value in trying to figure them out. The primary argument made by the WRAP authors and now the majority of OAuth 2.0 contributors is that signatures are hard and developers are stupid. This combination, they argued, is costing them developers.

To address this, they argued that the only solution is to remove signatures. I countered that instead of creating a new protocol, the companies complaining (primarily, Google, Microsoft, and Yahoo!) should invest in quality libraries and debugging tools.

My point was (and still is) that if you give developers value, they will fight to figure out the signatures. A couple of weeks ago Twitter discontinued their support for Basic authentication, and what these people said cannot happen, happened. All these developers figured out how to migrate their application to OAuth 1.0. That despite the lacking Twitter developers support, alleged bugs, and other complaints about Twitter’s implementation.

Conclusion

Don’t expect knights to battle dragons if your castle is empty. Twitter put a hot blond (or brunette) princess (or prince) in their castle, and their (API) knights fought the evil (signature) dragon and got their reward. Google and the rest of the big web providers with their useless offering of boring APIs left their castle empty.

Guess what! The kind of knights who come to fight dragons living in empty castle are there for the fight, not to do something useful. Yes, battling dragons is a bitch, but knights tend to forget that once they get their happy-ever-after. Give them an empty castle and they will do nothing but obsess about their battle scars.

Why does this matter? Because without signatures, there can be no secure discovery and less open web.

How to Enter a Standards Working Group

2010-09-06T06:16:57Z

As the OAuth specification editor, I am approached daily with comments, questions, ideas, and some very odd solicitations. It is quite fun. It seems that most people joining the work are determined to undermine their contribution by doing their best to alienate or insult the community, and often the editor.

This is not unique to OAuth – I have seen the same mistakes in many other places. Since working on an early draft of the OAuth 2.0 specification, I started collecting tips for newcomers. Here are some that would make your entrance into a standards working group much smoother and productive.

Don’t start with a proposal, start with a question.

Assume you don’t understand everything, especially if you have feedback or ideas for changes. Instead of jumping in with some new re-architecture of the entire protocol, start by asking why it is the way it is. Engage the community first with a question, and work your way to change requests based on the answer you receive. By asking a question, you allow for the possibility that whatever if bugging you has a valid reason.

Keep your message as short and to the point as possible.

You are an unknown so people will look for reasons to ignore your message. Most people don’t even open messages from newcomers. Instead, they wait to see if a thread develops. In other words, they wait until someone else puts in the extra work to deal with the new guy. If your message is short, it is more likely not to be ignored.

Don’t email the editor privately, asking for permission before asking the list.

You don’t need permission to ask questions or post to the list. All the specification editors I know (and certainly me) are usually busy and work in bursts. This means your email is most likely going to be moved to some queue of questions and feedback about the specification, to be processed right before the next editorial cycle. You are also wasting their time by forcing them to read your message twice (once directly and once on the list), and in many cases, repeat the discussion. This is why we have a public list – just join and post away.

Don’t prefix your message with I’m new here.

It’s never a good idea to start a relationship with an excuse. The only reason to put an “I’m new here” disclaimer on your message is if you were too lazy to do your homework, read group archives, read the full specification (and revision history), and think before writing. If you do all that, your contribution is as valid and relevant as that of anyone else. The working group regulars know you are new.

Focus on what you know best.

Pick an area where you have experience and provide feedback to show you know what you are talking about. Don’t start with a multi-page review of the entire specification. Instead, focus on the areas you are most knowledgeable about and demonstrate your skills and experience through the quality of your contribution. Don’t provide a summary of your career – saying you have many years of experience is the least effective way to get that message across. Let you work speak for itself.

Never start with an editorial suggestion.

Editors like to write, and they take pride in their work. They also have their own unique voice. Starting a relationship with a specification editor by criticizing their style is not a good way to get them to listen. Typos, mistakes, and other technical issues with the specification are always welcomed, but offering replacement text or ideas on how to say something differently will most likely be ignored, no matter how good the suggestions are. Wait for an editor to ask you to suggest text before sending in your own revision.

Start with a compliment, show good progress.

Pick something you think is good about the work, and highlight it. This isn’t rocket science, just common sense. Don’t compliment any one person in particular (like the editor or chair). Compliment the work and the community and do it in a way that adds value, like highlighting what is strong and good about the work, providing additional support.

Clearly state your role and introduce yourself.

Spend a few words to introduce yourself the first time you write something. Don’t just barge in – it’s rude. Also, standards are highly political, and in a political environment, people want to know who they are dealing with. No need for life story, just the basic facts (lurker, developer, representative for a big company). And don’t assume everyone know who you are just because you think you are important.

Do your political homework.

Identify the key participants and align yourself with those you agree with, showing you are well versed in the internal politics. Joining the discussion through agreement with a primary participant allows you to establish a work relationship with those most likely to be in a position to listen and impact. Agreeing with someone is usually better than disagreeing – it adds emphasis to an existing view, and doesn’t cost you political points going directly against someone else.

Make sure to be right if you are going to keep a thread going.

If you are going to stand your ground and keep an argument going, against the group consensus, make sure you know what you are talking about. It is perfectly fine to keep an issue going if you are unhappy with the resolution, but if you are new, this behavior is more likely to alienate the group than win your argument. If you are going to keep at it, make sure to add new facts and new examples to make your point.

Don’t explain basic standards principals.

Assume everyone knows what bike-shedding, design by committee, or KISS are. Don’t lecture the working group about process and other philosophical ideals. For most of the people involved, this is probably not the first standard.

Don’t prefix more messages with “I’m still new here”.

You didn’t need to state it the first time, so please don’t keep repeating you are (still) new. Yes, we got it, you are new here.

All This Twitter OAuth Security Nonsense

2010-09-02T22:53:58Z

In a wordy article that could have been much shorter and a lot less sensational, Ryan Paul of ArsTechnica throws mud mostly at Twitter, but saves plenty to throw at OAuth. Unfortunately, Ryan Paul (who clearly is a smart guy) is heavy on the accusation but light on the arguments. Typically, I would go over this article an item at a time, but I’m right in the middle of draft 11 of OAuth 2.0 which is a much better use of my time. If you want to read a great rebuttal, Ben Adida’s response (as always) is a great read.

The OAuth standard has many significant weaknesses and limitations. A number of major Web companies are collaborating through the IETF to devise an update that will fix some of the problems, but it’s still largely a work in progress. The current version of the standard—OAuth 1.0a—is an inelegant hack that lacks maturity and fails to provide clear guidance on many critical issues that are essential to building a robust authentication system.

First, the current version of the standard is RFC 5849. The RFC not only changed a lot of the terminology, highlighted the known security limitations, and has completely new prose, it also explicitly clarified at least one of the author’s issues regarding the handling of timestamps (which most companies don’t even bother to check anyway).

My favorite silliness from the article, when discussing the lack of specification details about using client secrets in installed applications:

Part of the problem is that the specification doesn’t provide much guidance about what implementers should do instead, which has forced them to improvise. Facebook and Google Buzz have both come up with reasonable solutions and offer desktop-appropriate OAuth authentication flows that do not require a secret key or require the end user to go through a complicated copy/paste process.

The specification is very clear (as the article quotes) – don’t use client secrets in installed applications! The reason why the specification doesn’t say much more is because there is no solution. It does not exist for a distributed application unless you issue a different secret to each installation. To say that Facebook and Google came up with reasonable solutions is pure sensationalism. What is their reasonable solution? They don’t use client secrets. In other words, they do exactly what the specification says.

The article does bring up important points implementers should pay attention to when using OAuth, such as the secrecy of their client credentials, the exact details of their user experience, how they authenticate the user (cookies, etc.), and an overall awareness to phishing. But OAuth, just like other security protocols, is designed to be implemented by security experts. In addition, there are simply no widely available solutions for many of these problems.

If Twitter uses the client secret in installed application for anything other than gathering statistics, well, they should reconsider. But it’s not like they have other alternative. That’s the only valid “news” the article has to offers.

It’s easy to throw mud without getting dirty with making a fully baked technical argument – and it makes for a fun read too. But when it comes to a widely deployed security protocol, scoring page views by scaring readers about security is not fair game. It is always valuable to highlight OAuth’s weakness, but in context, with the right security risk analysis, and with clear comparison to other alternatives.

I expect more from ArsTechnica.

An Introduction to OExchange

2010-06-10T06:19:05Z

OExchange is a newly-introduced protocol stack that allows users to share URL-based content with any service on the web. It covers posting links to social networks as well as sending content to things like online translation and printing services.

The protocol — driven by the folks at Clearspring (where I work) with the support of a long list of online services — builds on several existing open web specifications. It is backed by an open development list, tools for developers, and lots of additional resources.

The Problem

Lots of services accept URL-based content — social networks, news and bookmarking sites, communication tools, long-tail forums, translation utilities — and more appear every day. Why do blogs only allow users to share to the top few social networks? Why can’t users engage with more personal, relevant, dynamically-discovered options? Why are sharing tools and publishers still “integrating” with services individually?

As I’ve written before, users should be able to send content to any of an unbounded number of services. Sharing features should dynamically adapt to support the communities of interest users actually participate in and the tools they use, whatever platforms they are built on, and whether or not they are well-known. Services should be able to receive content from anywhere, in a common way. Basic interoperability benefits the web at large — services, users, and content providers — and we see a tremendous gap when it comes to simple URL exchange.

The Solution

A common protocol means more content for services, easier integration for tools, more personalization for users, and more traffic back to publishers. OExchange defines the technical solution in three areas (take a look at the complete spec for all the gory details).

1. A way for services to receive content

Known as an “Offer” in OExchange terminology, this method maps onto the dominant model already deployed widely today. Specifically, for a service to be able to accept content via OExchange, it exposes an endpoint of the form:

http://www.example.com/share.php?url=http://hueniverse.com

This endpoint (which can be located at any path) accepts a set of standard arguments. Only one of these arguments, the url, is required. Anyone can send content to this service by sending the user’s browser to this endpoint. The receiving service controls any relevant authentication as well as other aspects of the user experience — the service is the one that defines what actually happens once the user sends their URL there.

A common method removes any and all service-specific integration requirements — all supporting services can accept content in the same way.

2. A way for services to make themselves discoverable

By building on top of the established XRD and host-meta specifications, OExchange makes it possible to integrate with services that you, as a content publisher or sharing tool, didn’t even know about at development time.

A service defines everything about itself in a structured and machine-readable document format (known as the OExchange Target XRD), and anyone that wants to integrate with it can easily locate this document using established means (such as host-meta references).

This makes it possible to determine that a service exists, and how to send content to it, dynamically as opposed through manual integration. The full discovery flow is outlined in the spec.

3. A decentralized, user-centric model for expressing preferred services

We are big believers in the promise of personal discovery. In OExchange, we recommend the use of WebFinger and, eventually, XRD Provisioning, to let a user publish a set of services they actually care about, in the form of service XRD URLs. The key idea here is that it should be possible to look up the set of link-accepting services (each expressed as machine-readable documents) appropriate for a given user. Tools can inspect this list and present the right options to the user at runtime.

Importantly, this list may include services the tool doesn’t actually know about — it can use the basic OExchange discovery mechanism to then figure out how to send content to this service. For example, imagine visiting a blog and being presented with not only the option to share content to the mainstream social networking service on which you have an account, but also the long-tail service no one but you and your 5 friends has heard of.

Do we need another protocol?

There is quite an open web capability stack available at this point, and OExchange leverages established protocols extensively. That said, there are a few things missing to accomplish certain domain-specific use-cases. For example, though there are any number of relatively domain-specific protocols that can be used to send content from one service to another, there are none which can offer a simple interoperability layer without introducing substantial requirements on the implementor. Its our belief that these requirements can counteract the benefit of the additional content to the service, and hamper adoption.

For anything that OExchange is newly-defining, it therefore keeps the recommendations as simple as possible, and actually takes pains to map onto predeployed models. Really, OExchange is as much a packaging of existing protocols into a specific use-case stack as it is a new protocol.

What next?

The strategy we’ve taken with the specification is to first codify an existing practice and be compatible with a wide range of existing deployed services, then to add a new service discovery capability on top of that, then to expand from that point into additional protocol actions and modes. In this vein, along with encouraging adoption and working to support the stack in the products we’re all building, we’re now beginning to focus in earnest on the additional capabilities. One often-cited example is an exchange mode that can take place without popping a new browser window/tab.

If you’re interested in getting involved in the development of the spec, please join the discussion. If you just want to get going on implementing it, check out the quick start guide.

We’re pretty excited about a more interoperable web, thanks for the chance to outline it!

How the Open Community Can Beat Facebook

2010-06-08T07:38:40Z

Well, the open community can’t beat Facebook.

But companies using open technologies can – by building better products. Outside the echo chamber of web standards fanatics, the vast majority of web users don’t care about how the web works. They care about their user experience, where their friends are, and when something goes wrong, protecting their privacy.

When I read about Google Buzz (and other open-based products), it is repeatedly described as the open alternative to Facebook. Does this information help me (as a consumer) make a better decision about which product to use? No. That’s like telling the average cell phone buyer that the difference between the iPhone and Android is that the latter uses an open source operating system. When it comes to selling phones, Google relies on their search reputation and brand, not the openness of their platform.

Getting consumers to use your products, like any other retail interaction, requires offering something useful that is better than other alternatives. It is true that sometimes a backlash against one company leads consumers to switch to someone else, but they don’t “vote for the new guy”, they “vote out the old guy”. If users leave Facebook to use Google, it is not a victory for Google – it is a loss for Facebook.

When it comes to showing the value in open technology, very few efforts can show how being open makes products better. Even if OpenID solved all its problems, found a less offensive solution to the NASCAR problem, got providers certified and trusted, provided a legal framework for managing liability, educated consumers, and actually worked, it will still fail without the wealth of data offered by Facebook.

Why should publishers (content and service providers) choose a solution that doesn’t deliver actual consumer value?

In an attempt to address this, the OpenID community has been looking for ways to compete with Facebook. The OpenID/OAuth Hybrid proposal was one approach. Adding rich profile data was another (in the conceptual proposal for OpenID Connect). But these are all focused on enabling technologies, not products. Even if there was a complete open solution for every Facebook feature, it would still not offer a compelling value proposition because without actual data behind it, it is nothing but empty containers.

If Facebook asked me, I would recommend using open technologies because it is good for business (when available and applicable). But to everyone else I would recommend focusing more on the product and less about the openness of the platform. Open is certainly a selling point in the enterprise market, but it is not in the consumer market.

Two years ago the big fight was against the “walled-gardens” and user data, now it is about open standards. It didn’t make a difference back then (users didn’t care) and it won’t make one now. Facebook didn’t change their data policies because of what users wanted – they changed it because of what publishers demanded, and the publishers asked for data, in whatever shape or form Facebook wanted to give it.

The reason why the newly proposed OpenID Connect protocol is actually promising is that it focuses on mobility instead of federation. Instead of trying to build a fully distributed and federated identity framework, the proposal uses OAuth 2.0 to build vendor-specific identity solutions that are all implemented the same way. By allowing publishers to move from one compliant vendor to another, it lays the groundwork for future federation and distribution.

In other words, the fact that it doesn’t embrace discovery at its core, but starts with reliance on client registration and vendor specific relationship is an assets because it guarantees better products with built-in mobility. That mobility will allow publishers to take their business elsewhere if they don’t get the data and services they want.

Good technology enables better products. Being open is just the cherry-on-top.

Then of course, there is the other option: if you can’t beat them, join them.

XAuth – a Terrible, Horrible, No Good, Very Bad Idea

2010-06-08T05:27:42Z

A few weeks ago, a handful of web companies lead by Meebo and Google (with moral support from Yahoo!) announced their support for a new protocol called XAuth. The idea is very simple and seemingly appealing – create a sort of shared-cookie service for sites to use to store and find which identity providers a user prefers, solving the OpenID NASCAR problem. It is a similar idea to existing commercial products such as JanRain’s RPX.

I’ve heard about this proposal a few months ago and have been rolling my eyes ever since. Why? Because this is – to borrow from one of my son’s favorite book – a terrible, horrible, no good, very bad idea. It is a dangerous and over simplified hack aimed at solving a complex problem – how to manage online identities and improve the usability of distributed identity providers.

The XAuth proposal is a privacy and security nightmare. It relies on the use of a single, shared domain name which is currently under the control of one company – Meebo. While I have nothing against Meebo, other than proposing and promoting this, I certainly don’t trust it to manage the web’s identity preference layer. I heard suggestions of handing control over to the OpenID Foundation, but I don’t trust it either. If it is controlled by a single entity, it fails the most basic test of distributed identity services.

In addition, XAuth is a cookie-like mechanism that suffers from all the same problems, and as such, is an easy target for abuse and manipulation. And guess what – it is an opt-out mechanism per browser.

This is a misguided attempt to solve a problem browser vendors have failed to address. It is true that getting browser vendors to care about identity and innovate in the space is a huge challenge, but solving it with a server-hosted, centralized solution goes against everything the distributed identity movement has tried to accomplish over the past few years.

As for the name, I resent the association between XAuth and the OAuth brand, using a cloned logo and a very confusing name. This is especially true considering it has nothing to do with OAuth, and everything to do with OpenID. And by the way, the name is already taken.

I think I’ll move to Australia.

Update: This post sparked a response from Google’s John Panzer (someone I highly respect, and respectfully disagree with) as well as an interesting thread on the OpenID list. A couple quotes I complete agree with:

From Peter Watkins:

The current XAuth implementation has sites using IFRAME elements to access the XAuth service/JS code. Web browsers send Referer headers with IFRAME, so whoever runs xauth.org is in a position to see information about what Extender and Receiver sites a user accesses. Currently auth.org has pretty good settings — cache control headers telling browsers they can cache the page for a week. But that could change. Move responsibility into the browser and that problem is solved.

Also, xauth.org could start delivering JS code that reports information to the xauth.org mothership in addition to simply “working”. Say the local government tries to compel xauth.org to deliver additional code to specific IP addresses (not that Google has *ever* had any trouble with any government legal pressure, right?). xauth.org could deliver pristine, trustworthy JS to everyone else. How would the government targets, let’s say political activists maybe, be able to tell their privacy was being subverted? Move the function to the browser and that hole is closed.

This is by no means a comprehensive list (shoot, it’s been less than 24 hours since I startd reading up on XAuth), but I think it’s enough that you can’t say there are no privacy issues that going to an in-browser model would solve no privacy issues.

From Philip Hallam-Baker:

As often happens in these debates, we have a proposal that has an acknowledged issue that we are being told isn’t an issue because the developers don’t see it as an issue.

I really take offense when I raise an issue and someone says ‘that does not matter to anyone’ or ‘that issue has been dealt with’. The one issue that I have never found it difficult to get the industry to agree on is the necessity of ensuring that no party gains a proprietary leverage in a communication protocol.

I don’t see how the promise that the issue will be fixed in future changes anything. Either the centralization is easy to eliminate from the protocol or it isn’t. And if it is easy to eliminate then why introduce it in the first place?

The starting point for identity in my view is that I have to entirely own my name. There cannot be any entity that can use the investment I make in my name to extract rent at a future date. No corporation, no not-for-profit, no non-profit, no industry group. Nothing.

(Image adapted from the book “Alexander and the Terrible, Horrible, No Good, Very Bad Day” by Judith Viorst, illustrated by Ray Cruz.)

Open vs. Fast, Good vs. Evil, Google vs. Facebook

2010-05-28T18:34:02Z

The landscape of the community-engineered social web, the one based on open technologies, has changed dramatically over the past few months. If you took a year off and just came back, you would probably not recognize it at all.

The movement that started with protocols such as OpenID, OAuth, and Activity Streams, is now mostly gone. All the cool kids got grownup jobs and the market is back again driven by a small number of corporations. In fact, it is so small it can be counted on two fingers. A year ago, a meeting with Chris Messina, David Recordon, Joseph Smarr, Monica Keller, Will Norris, Luke Shepard, and John Panzer represented 7 different organizations or communities – a well-balanced mix of big and small, corporate and independent.

Today it’s just Facebook and Google and that has significant implications. But when examining how these two companies engage in the development of open technologies, the findings are quite surprising. On the product side, Google is famous for their openness while Facebook is notorious for their closed garden. But when it comes to their community engagement, these two giants behave in a rather reverse fashion.

Open technology is slow by definition.

That’s assuming you accept my definition of Open Technology:

Developed in the open with full transparency
Open process for anyone to participate freely
Everyone is free to implement
Decisions are made based on technical merit

Open technology doesn’t have to happen in a standards body or format open source organization, but important work usually does; because if it’s important, there are simply too many people collaborating to be successful without a well-defined process and governance. Successful open technology acts very much like a proprietary one – they both tend to block or delay new innovation. As soon as something becomes successful, it poses high risk to those absent from the process.

OAuth 1.0 created a myth that it is possible to develop open technology fast. But in reality, OAuth 1.0 took a little over a year, and that’s with a tiny, well-aligned group of people almost exclusively from one town, with very little at stake. The numbers are even less impressive if you include getting to revision A as part of the 1.0 lifecycle.

OpenID had a very similar experience where version 1.0 (really 1.1) took little time (mostly created by two people) while version 2.0 took a very long time. Deciding what the next version of OpenID looks like seems to take even longer.

Two years ago I was one of the leaders of this movement. It was the main driving force behind creating the Open Web Foundation (i.e. a lightweight legal framework suited for small, community driven projects). The problem with this approach is that it ignores why and how companies develop open technology, and the real reasons why it takes time.

People new to standards like to accuse the excessive process and legal framework for the time it takes to produce a standard. But this argument is simply not grounded in reality. For example, XRD 1.0 which is quickly becoming one of the building blocks of the social web took over 2 years to mature from XRDS through XRDS-Simple to XRD. During those 2 plus years, the OASIS process (where the work was done) did not slow us down by more than a week or two. What slowed XRD’s development was lack of feedback, review, and editorial time. But even these factors were not the primary reason.

There are two reasons why specifications take time: building consensus and reaching maturity. Consensus time is usually a function of the group size. The bigger the group the longer it takes. Maturity however, is a function of taking intentional breaks during the process to let the technology sink in, and allow time for experimentation. Maturity is what differentiates a useful technology from an essential one.

If I listened to the many calls to get XRDS-Simple done two years ago because people wanted to ship stuff “now“, we would have ended with broken discovery architecture and a complex format that no one really needed. On occasion, these demands for finalizing specifications took the form of subtle threats to develop competing proposals. Getting the right people to read specifications takes time, and it often means waiting for the use cases to catch up with the vision.

If you compare the initial proposal of site-meta with the final RFC for Well-Known URIs, you immediately notice a remarkable transformation which was the result of a yearlong discussion and collaboration across three standards bodies. My discovery work follows the same pattern, from the initial OAuth Discovery work to the current, much simplified host-meta proposal (and the retirement of the LRDD stand-alone specification).

Like anything else, early adopters have to accommodate this reality, and it can often lead to frustration. The Open Web Foundation is one example for how this frustration with standards bodies materialized into an (unsuccessful) attempt to create a whole new framework. While the foundation was very successful in creating a useful legal license, it was not significant enough to get new technology out faster. Ironically, it made it much easier for companies to develop new technologies internally, the open them up when done.

In order to understand the forces that drive the development of open technologies, we must first examine how companies address these issues. There are generally three categories of companies when it comes to interaction with open technology: conservative, agile, and delayed-open. It is important to understand that these categories are specific to how companies interact with open technology, and are not an indication of how the interact in the marketplace.

Traditional companies are rarely first to adopt new technology, wait for final versions before implementing, and actively participates only when a specification takes a wrong turn. They are usually absent from the process or lurk around, but engage with some form of indirect support (such as sponsoring events or supporting the community). Traditional companies tend to focus on building business relationships with other similar companies and establish back-channel alignment, which enables them to let others represent their interests. Yahoo! and Twitter are good examples of traditional companies.

Agile companies live on the cutting edge, deploying early drafts and are not afraid to change quickly. They are usually very hands on, often leading the development efforts, and maintain some level of control over the process, which helps reduce risk by having better understanding of the proposal’s stability. Agile companies are usually very small where being agile provides a necessary edge, or very big where they can afford to take more risk and experiment because of their market dominance. Facebook and (the now defunct) Pownce are good examples of agile companies.

Delayed-open companies are very selective in what they choose to participate in, and are rarely involved in efforts they do not control. The basic strategy of delayed-open companies is to develop as much as possible internally and confidentially. They only share their work when they are ready to ship a product or when they are confident in the stability of the technology. While these technologies end up being open (typically by opening their development to final enhancements, or by contributing them to a standards body), by the time they are opened up, very little can really change. Google and Microsoft are good examples of delayed-open companies.

Companies looking to engage in the development and utilization of open technologies must find the right balance that fits their culture and market needs. Google and Facebook provide important case studies on how companies approach open technology from opposite ends of a given market. When it comes to social web applications, Facebook is the undisputed market leader, while Google is one of the (hardest trying but) least successful players.

Both companies recently made high profile hiring moves, grabbing every free-agent open technologist in the space. Facebook hired David Recordon and Monica Keller (joining Luke Shepard and others), while Google hired Chris Messina, Joseph Smarr, and Willl Noris (joining John Panzer, Brad Fitzpatrick, and others). But their reasons for doing so are fundamentally different, dictated by their involvement category.

Facebook, with time on their side, hired people to help open up their technology and use their market dominance to influence and lead new efforts. Their position allowed them to deploy an early draft of OAuth 2.0 in a highly publicized fashion, and to promise keeping their production code up to date as the specification matures. To accomplish that, their engineers engaged early and intensely, contributing the first draft and running code. The more Facebook invests in open technologies, the longer it takes these technologies to mature (due to their sudden importance and popularity), but with a higher likelihood of maturity.

Google on the other hand, hired top caliber talent for very different reasons. Google is significantly behind Facebook and cannot afford to take years (or even months) for new technologies to mature. They need to build products and ship them quickly, and that requires much more control than is available in open development. By hiring highly respected individuals like Chris Messina and Joseph Smarr, Google is hoping to get away with doing more work delayed-open. Salmon, PubSubHubbub, WRAP, and their OpenID extensions are all examples of past technologies developed successfully using this method.

While Facebook shipped an early draft of OAuth 2.0, Google shipped WRAP. Both promised to ship the final OAuth 2.0 protocol.

It is hard to say which category ends up making the biggest contribution to the development of open technologies. Google is clearly a leader in adopting open technology in general, but the way in which they get comfortable doing so isn’t always the most open or polite. Google’s style helps get more free technology faster, but it also makes is much harder for other companies to participate when the foundation is established. Google is leading an Open war, and in war, there are often innocent casualties.

There is a direct correlation between a company’s ability to control the process and their embrace of the technology. Facebook came in late to the WRAP party, leading them to favor the (at the time) less prominent and less successful OAuth 2.0 effort at the IETF (while Google did their very best to derail the IETF effort). In the same spirit, Google’s bear hug of HTML5 came only after they guaranteed their strong control over the process by hiring the specification editor (for the sole purpose of getting HTML5 done fast).

The manifestation of these two approaches is sometimes ironic. Facebook who is not known for their embrace of open protocols and technology, is the one enabling open communities to take their time and spend a little longer to get the details right. On the other hand, Google who is considered the biggest champion of openness is doing their best to push efforts to a quick conclusion, strongly aligned with their immediate needs.

While this analysis includes a slight bias in favor of the contribution of agile companies, delayed-open companies play an important role in creating alternatives, not possible by purely open means alone. Facebook only turned the page and joined the open community when it felt the open community no longer posed a threat to their dominance, and their behavior depends greatly on the individuals leading the effort. The Facebook influence at the hands of David Recordon (who at this point is by far the most influential voice in advocating open technologies) is an extremely constructive force. However, it is clear that without Google chasing their tail and portraying them as closed, Facebook will be much less motivated to open up.

It is also important to remember the significant contribution of traditional companies. They provide the final seal of approval for new technologies and are the key to mass adoption. They are also critical in getting enough community sponsorship to allow work to continue.

At the end we rely on a few individuals to do the right thing. On the Google side, we rely on people like Chris Messina and Joseph Smarr to know where to draw the line between delayed-open and fully-open. Because delayed-open leave very little for the community at large to influence, they must find the right balance between pragmatic time-to-market and forcing their own ideas on the rest of us. When Chris and Joseph joined Google, I predicted exactly this kind of shift. I have confidence that their work will continue to be innovative and inspiring, but the silence and disengagement coming from the Google team over the past six months is a real cause for concern.

Introducing OAuth 2.0

2010-05-15T07:13:43Z

Two weeks ago, the IETF OAuth Working Group published the first draft of the OAuth 2.0 protocol. OAuth is a security protocol that enables users to grant third-party access to their web resources without sharing their passwords. OAuth 1.0 was published in December 2007 and quickly become the industry standard for web-based access delegation. A minor revision (OAuth 1.0 Revision A) was published in June 2008 to fix a security hole. In April 2010, OAuth 1.0 was published as RFC 5849.

OAuth 2.0 is a completely new protocol and is not backwards compatible with previous versions. However, it retains the overall architecture and approach established by the previous versions, and the same introduction (from the Official Guide to OAuth 1.0) still very much applies.

Many luxury cars come with a valet key. It is a special key you give the parking attendant and unlike your regular key, will only allow the car to be driven a short distance while blocking access to the trunk and the onboard cell phone. Regardless of the restrictions the valet key imposes, the idea is very clever. You give someone limited access to your car with a special key, while using another key to unlock everything else.

As the web grows, more and more sites rely on distributed services and cloud computing: a photo lab printing your Flickr photos, a social network using your Google address book to look for friends, or a third-party application utilizing APIs from multiple services.

The problem is, in order for these applications to access user data on other sites, they ask for usernames and passwords. Not only does this require exposing user passwords to someone else – often the same passwords used for online banking and other sites – it also provide these application unlimited access to do as they wish. They can do anything, including changing the passwords and lock users out.

OAuth provides a method for users to grant third-party access to their resources without sharing their passwords. It also provides a way to grant limited access (in scope, duration, etc.).

For example, a web user (resource owner) can grant a printing service (client) access to her private photos stored at a photo sharing service (server), without sharing her username and password with the printing service. Instead, she authenticates directly with the photo sharing service which issues the printing service delegation-specific credentials.

The new draft represents a yearlong discussion around goals and requirements for the protocol with participants from a wide range of companies including Yahoo!, Facebook, Salesforce, Microsoft, Twitter, Deutsche Telekom, Intuit, Mozilla, and Google. OAuth has long been the poster-child of rapid open technology adoption, and 2.0 is no exception – Facebook and Twitter already announced support even before the first draft was officially published.

Why a New Version?

OAuth 1.0 was largely based on two existing proprietary protocols: Flickr’s API Auth and Google’s AuthSub. The result represented the best solution based on actual implementation experience. With over 3 years of experience working with the protocol, the community learned enough to rethink and improve the protocol in three main areas where OAuth 1.0 proved limited or confusing:

Authentication and Signatures

The majority of failed OAuth 1.0 implementation attempts are caused by the complexity of the cryptographic requirements of the specification. The fact that the original specification was poorly written didn’t help, but even with the newly published RFC 5849, OAuth 1.0 is still not trivial to use on the client side. The convenient and ease offered by simply using passwords is sorely missing in OAuth.

For example, developers can quickly write Twitter scripts to do useful things by using their username and password. With the move to OAuth, these developers are now forced to find, install, and configure libraries in order to accomplish what was before possible with a single line of cURL script.

User Experience and Alternative Token Issuance Options

OAuth includes two main parts: obtaining a token by asking the user to grant access, and using tokens to access protected resources. The methods for obtaining an access token are called flows. OAuth 1.0 started out with 3 flows: web-based applications, desktop clients, and mobile or limited devices. However, as the specification evolved, the three flows were merged into one which (on paper) enabled all three client types. In practice, the flow works fine for web-based applications but provides an inferior experience elsewhere.

As more sites started using OAuth, especially Twitter, developers realized that the single flow offered by OAuth was very limited and often produced poor user experiences. On the other hand, Facebook Connect offered a richer set of flows suitable for web applications, mobile devices, and game consoles.

Performance at Scale

As large providers started using OAuth, the community realized that the protocol does not scale well. It requires state management across different steps, temporary credentials which are more often than not discarded unused, and typically requires issuing long lasting credentials which are less secure and harder to manage (and synchronize across data centers).

In addition, OAuth 1.0 requires that the protected resources endpoints have access to the client credentials in order to validate the request. This breaks the typical architecture of most large providers in which a centralized authorization server is used for issuing credentials, and a separate server is used for API calls. OAuth 1.0 requires the use of both set of credentials: the client credentials and the token credentials which makes this separation very hard.

So What’s New?

The following is a subset of the new features available in OAuth 2.0:

6 New Flows

User-Agent Flow – for clients running inside a user-agent (typically a web browser).
Web Server Flow – for clients that are part of a web server application, accessible via HTTP requests. This is a simpler version of the flow provided by OAuth 1.0.
Device Flow – suitable for clients executing on limited devices, but where the end-user has separate access to a browser on another computer or device.
Username and Password Flow – used in cases where the user trusts the client to handle its credentials but it is still undesirable for the client to store the user’s username and password. This flow is only suitable when there is a high degree of trust between the user and the client.
Client Credentials Flow – the client uses its credentials to obtain an access token. This flow supports what is known as the 2-legged scenario.
Assertion Flow – the client presents an assertion such as a SAML assertion to the authorization server in exchange for an access token.

Native application support (applications running on a desktop or mobile device) can be implemented using many of the flows above.

Bearer tokens

OAuth 2.0 provides a cryptography-free option for authentication which is based on existing cookie authentication architecture. Instead of sending signed requests using HMAC and token secrets, the token itself is used as a secret sent over HTTPS. This allows making API calls using cURL and other simple scripting tools without having to canonicalize the request and sign it.

Simplified signatures

Signature support has been significantly simplified to remove the need for special parsing, encoding, and sorting of parameters. It also uses a single secret instead of two.

Short-lived tokens with Long-lived authorizations

Instead of issuing a long lasting token (typically good for a year or unlimited lifetime), the server can issues a short-lived access token and a long lived refresh token. This allows clienta to obtain a new access token without having to involve the user again, but keeps access tokens limited. This feature was adopted from Yahoo!’s BBAuth protocol and later its OAuth 1.0 Session Extension.

Separation of Roles

OAuth 2.0 separates the role of the authorization server responsible for obtaining user authorization and issuing tokens from that of the resource server handling API calls.

When is it coming out?

OAuth 2.0 is currently in development by the IETF OAuth Working Group. The latest draft is considered stable but with many features being changed or added. Partial support is already available from Facebook and Twitter. The final specification is expected by the end of the year.

JRD, the Other Resource Descriptor

2010-05-24T23:21:54Z

(Yes, that was a LRDD-inspired pun.)

XRD 1.0 is the result of 5 years of community development and actual deployment experience. It represents the most concise, yet extensible way to describe web resources using well understood constructs such as links. It uses XML as its extensible backbone, enabling protocols to extend pretty much every element as needed. For a long time, XML was the source of XRD’s (and its predecessor, XRDS’s) extensibility.

But XML is no longer what puts the ‘X’ in XRD. XRD utilizes the Web Linking framework for accomplishing cross-format interoperability of link relation types, as well as URI-namespaced key-value properties (for both the resource properties and links). XRD 1.0 is extensible without having to define new elements or attributes. It is accomplished simply by using its <Link> and <Property> elements as-is.

There might be cases where protocols will want to extend the schema, and that is where the XML foundation comes in handy. In addition, XML provides a mechanism for digital signature (which has as many critics as fans), and a clean way to add new elements without name collisions.

However, XML is a heavy format, and is no longer the format of choice in many new web protocols. JSON, a simple but powerful serialization format is getting more and more popular. OAuth 2.0 is likely to support JSON, following its adoption by popular services such as Twitter and Facebook as the default format of their APIs.

Introducing JRD

JRD, pronounced “Jared” and stands for JSON Resource Descriptor, is a JSON-formatted XRD document. It takes the XRD schema and converts it to a JSON structure, giving up some XML-based features, but gaining simplicity and adaptability to JSON-centric protocols and applications. JRD is based on a few simple rules:

XRD is the canonical representation; JRD is meant as an alternative format offered in addition to XRD.
JRD does not include support for new extension objects (XRD elements); it is meant for cases where the core XRD elements are sufficient.
XRD can be converted to JRD; a lossless conversion of JRD back to XRD is not always possible.
Obtaining JRD is designed as a special case request for an XRD using the HTTP Accept header or format request parameter; JRD is available where XRD is available.

In other words, services should continue using XRD as the primary format for their descriptor documents, and XRD should remain the default format for resource descriptors using the XRD schema. However, services may also support obtaining a JRD version of the document by allowing clients to specify a preference for JSON instead.

JRD supports the following XRD elements: <Subject>, <Alias>, <Expires>, <Property>, <Link>, and <Title>, as well as the attributes of the <Link>, <Property>, and <Title> elements. It does not support digital signatures.

Since nothing explains formats better than examples, the following fully-featured XRD document:


<?xml version='1.0' encoding='UTF-8'?>
<XRD xmlns='http://docs.oasis-open.org/ns/xri/xrd-1.0'>

  <Subject>http://blog.example.com/article/id/314</Subject>
  <Expires>2010-01-30T09:30:00Z</Expires>

  <Alias>http://blog.example.com/cool_new_thing</Alias>

  <Property type='http://blgx.example.net/ns/version'>1.2</Property>
  <Property type='http://blgx.example.net/ns/ext' />

  <Link rel='author' type='text/html'
        href='http://blog.example.com/author/steve'>
    <Title>About the Author</Title>
    <Title xml:lang='en-us'>Author Information</Title>
    <Property type='http://example.com/author/role'>editor</Property>
  </Link>

  <Link rel='author' href='http://example.com/author/john'>
    <Title>The other guy</Title>
  </Link>

  <Link rel='copyright' href='http://example.com/copyright' />
</XRD>

Is represented by the following JRD document:


{
 "subject":"http://blog.example.com/article/id/314",
 "expires":"2010-01-30T09:30:00Z",
 "aliases":
 [
   "http://blog.example.com/cool_new_thing"
 ],
 "properties":
 {
   "http://blgx.example.net/ns/version":"1.2",
   "http://blgx.example.net/ns/ext":null
 },
 "links":
 {
   "author":
   [
     {
       "type":"text/html",
       "href":"http://blog.example.com/author/steve",
       "title":"About the Author",
       "titles":
       {
         "en-us":"Author Information"
       },
       "properties":
       {
         "http://example.com/author/role":"editor"
       }
     },
     {
       "href":"http://example.com/author/john",
       "title":"The other guy"
     }
   ],
   "copyright":
   [
     {
       "href":"http://example.com/copyright"
     }
   ]
 }
}

JRD does not support extension XRD elements or attributes. Future specification with use cases for extending JRD can define how to avoid name collisions (such as using a public wiki, registry, or namespaces).

Obtaining a JRD Document

In general, endpoints should not return a JRD representation by default. They should return an XRD document. Clients looking for a JRD representation make their preference known by using the HTTP ‘Accept‘ header with the ‘application/xrd+json‘ media type, or include the ‘format‘ parameter with a value of ‘json‘ in the request (or both).

For example, requesting a host-meta using JRD:


> GET /.well-known/host-meta?format=json HTTP/1.1
> Host: example.com
> Accept: application/xrd+json

Updates

5/24: Removed extensibility, changed array names to plural, turned links into a hashed list using rel values. Added title and titles object for the simple and complex use cases. Changed mime type to application/xrd+json.

5/14: Changed ‘namespace’ to ‘ns’ and ‘title’, ‘ns’, and ‘properties’ from array of objects to an object.