hueniverse

Hiring Engineers, a Process

2013-04-24T06:00:25Z

Disclaimer: The process outlined in this post reflects my personal approach. Please consider this as a helpful insight into what it takes to get a hiring recommendation from me. As always the law and corporate policy applies.

This post has three purposes. First, to save me the need to explain every time how I interview and hire people. Second, to inspire others to break away from the conventional and ineffective hiring process most companies use. It’s a process that fails to identify non-conforming great talent. And third, when the time comes for me to look for my next adventure (and no, I’m too happy where I am – I’m trying to hire you!), I can point hiring managers here to know how I’d like to be treated.

TL;DR

If you don’t have time to read this, we are not a good fit.

What Not to Do

In 2006 I was looking for a new job on Wall Street. I wasn’t happy at my job, but not yet ready to do my own startup. At the time I was a VP of engineering at Citi managing about 35 people. I was looking for a similar role with a smaller team and more hands-on opportunities. I was not looking for a primarily coding role. I got invited to interview with Credit Swiss for a lead role. When my first interviewer spent half an hour asking me about the order of initialization of static variable in C++, I played along. But when the second guy came in, put a laptop in front of me and said “We’re going to write some code now”, I got up and said “I’m going home now”. I didn’t get that job.

Later that year, I had an interview at Bloomberg. The interviewer greeted me at the entrance, then purposely walked me through their showy floors and world-famous curved escalator as if that’s going to motivate me. We’ve spent the first 10 minutes going over my financial products experience and then followed with 20 minutes asking me questions specifically about everything I just said I don’t have experience with. The next interviewer asked me about my coding experience which at the time was all C++, then followed up with only C questions. The last interviewer asked a riddle (“4 members of U2 are trying to cross a bridge”) which I’ve spend the entire half hour passive-aggressively not trying to solve. I didn’t get that job.

In 2009 I was approached by Google at the recommendation of some people I collaborated with. The conversation didn’t go very far when they made it clear that they will not be able to tell me what I will be working on, and more importantly, who I will be working with and for, until I start. I had serious doubts about passing their interview process, or even fitting in at Google, but I was willing to give it a try. What I wasn’t willing to do is consider a job where the only known parameter is the name of the company. I didn’t get that job.

I’m purposely focusing on companies that got it all wrong. This is not at all about settling scores. Over the years I’ve interviewed at companies like Twitter and Facebook, which resulted in offers I turned down, and companies like Mozilla and LinkedIn which did not result in offers. The point is, I don’t judge the quality of the hiring process based on my personal experience or whether an offer was extended.

I judge it based on how much I enjoyed the process and respected the people on the other side.

I – Screening

The key to a successful hiring process is a good screening filter. I look for at least one of three things when considering engaging a candidate:

a record of open source, standards, or other public work,
a recommendation from someone I respect, or
a cold call with an attitude

I don’t phone screen. I don’t read resumes.

The first two are self-explanatory but apparently not obvious. Too many people rely on their resume for engineering roles. I don’t care about your resume. What I’m looking for is code to look at, specifications to read, or products to play with. Nothing showcases your skills better than actual code, especially code written to satisfy your own standards and not an employer.

Without a public record or a recommendation, I’m not likely to have much interest. Which means, you need to find a way to get my attention. Sending a one line email (or a tweet) asking for more information is not going to accomplish much. I’m looking for some attitude. If you want me to spend time learning about you without having easy access to a sample of your work, you need to make it fun for me.

The obvious ways are to engage me on something I care about. Google me – it’s not hard to figure it out. Send me comments about a blog post or an open source project I published. Tell my why I’m wrong, why my project sucks, how I can make it better, or why you want to work with me. If you put an effort into it, I will do the same.

II – Greeting

Great, you got my attention. Now I am going to get yours.

Our first call or meeting is going to be all about you. My job is to sell you the company, the team, and the role. The sole purpose of our first interaction is to make you want the job. To give you all the information you need to make an educated decision. After all, in the next step I’m going to make you work for it, so it’s only fair you have all the information you need before I waste your time.

III – Homework

If you made it here, it means your background is exactly what we want, and this is the job you are looking for. Now let’s write some code!

I suck at writing code on demand. I can’t really code outside my controlled environment. I got my chair, and desk, and big monitor, and IDE, and color scheme, and… you get the idea. I also do my best work when I get to “sleep on” a problem. Some people are great at writing code on a whiteboard. It’s a great interview skill to have. It’s also an idiotic skill to test for. I am also not a fan of pair programming which would be the only valid reason to live code during an interview. So no live coding.

Instead, I will present you with a few problems the team is working on right now. I’m not going to make shit up. These are all actual problems we need to solve or have solved recently. The advantages are many: you get to experience what the job is really like and we get to work with you on a problem we have been looking at recently. More often than not, these are pending items that we have not solved yet. Looking at our open issues on GitHub is a great way to get an idea what you might be tasked with.

The guidelines are simply – pick a problem and come back with something that best showcases your abilities. It can be code, a pull request, a written proposal, or an email analysis. Whatever you feel is the best way for you to shine. You can use any resources you want – the web, books, friends. You know, just like you do at work. To get an idea what past hires submitted, check out some of these proposals (some are actual answers from interviews posted with permission after a successful hire).

You also get as much time as you want. There is no deadline.

There is only one catch. Once you send your response back, we’re going to chat about it. We’ll challenge your analysis, your assumptions, your solution. We’ll try to put sticks in your wheels, introducing new constraints or share our experience if we already tried that. In other words, we’ll have an educated discussion about the problem, now that you had as much time as needed to study it. No surprises.

The homework follow up can be a 30m phone chat or part of the in-person interview. It mostly depends on your location, availability, and how blown away we are with your submission. On occasion we are left speechless.

Funny story. during my time at Citi, leading a C++ shop, I gave this homework to people: “Build a car in C++”. Most people created a set of classes and put them together to construct a car object with some methods and state. One guy sent this (actual submission!):

class Car {

private:

  void top () {
    printf("                         ________________________");
    printf("                        /             |          \\");
    printf("                       /              |           \\");
    printf("                      /               |            \\");
    printf("                     /                |             \\");
    printf("                    /                 |              \\");
    printf("                   /                  |               \\");
    printf("                  /___________________|________________\\__________");
    printf("     ____________/                                                \\");
    printf("    /                                                              \\");
    printf("   /                                                                \\");
  };

  void bottom () {
    printf("  /                                                                  |");
    printf(" /        ,---------,                     ,-----------,              |=====");
    printf("|        /  ______   \\                   /   ________  \\             |=====");
    printf("|       /  /      \   \\                 /   /        \\  \\___________/");
    printf(" \\_____/  /        \   \\_______________/   /          \\");
    printf("         /          |                     |            |");
    printf("        |           |                     |            |");
    printf("        |          /                      \\           /");
    printf("        \\         /                        \\         /");
    printf("         \\_______/                          \\_______/");
  };

public:

  void drive () {
    top();
    bottom();
  };
};

Needless to say, he was invited for an interview. He didn’t show.

IV – Interview

The goal of the in-person interview is to evaluate the following areas:

Cultural fit – are you going to be a good addition to the team and work well with others? Will we enjoy working with you? Will we want to go to a conference with you? Also, as a team with remote members, evaluate your compatibility with our style and process.
Adaptability – we work in a mixed environment where we have control over our systems but little control over the services we consume. Working in this environment, with many legacy systems isn’t for everyone.
Passion – everyone should have something they are passionate about. What are you? We’ll try to find yours and chat about it a bit. It doesn’t have to be super relevant. The goal is to see how deep you dive into topics that excite you. My 2 hours interview at Yahoo! included an hour discussion about water gardens. It was awesome.

The on-site interview is usually 2-3 hours long with 3-5 team members. I usually ask each interviewer to focus on one of the areas above. I also provide them with the following list of do’s and don’ts.

The DOs:

Know what you are interviewing for (you’d be surprised).
Begin the interview describing your role in the team and allow the candidate to ask questions.
Ask questions based on actual team experience.
Focus on architecture, design, and approach to problems more than the fine details of the solution.
Let the candidate explain the process if their answer isn’t what you were expecting – a wrong answer can still demonstrate solid thought process.
Any coding question must be a follow up on the homework submitted.
Make it fun. Find ways to make the candidate feel at ease and enjoy the conversation.
Leave at least 5 minutes at the end for them to ask questions and unwind. Ending poorly sets the tone for the next interview.
Offer them a drink or a restroom break.

The DON’Ts:

Don’t ask those who interviewed the candidate before you what they think. Form your own first impression.
No whiteboard coding or any live coding during the interview. Some people do well coding on the fly, most don’t. It is not a skill we care about. It also takes too much time that is better spent talking.
No riddles, logic, trick questions, or puzzles. This is not the SAT.
Don’t ask questions about stuff they claim not to know. This should be obvious but sadly its not to some people. Ask them about what they know and claim to be proficient in.
Talk, don’t interrogate.
Don’t read their resume during the interview. In fact, don’t bring their resume to the interview at all. If you did not read it ahead, focus more about technology in general and our work. It is obvious when someone is interviewing you who has no clue who you are.
Don’t ask one long question in multiple parts. If they get the first part wrong, it sets a bad tone for the rest of the interview. Give them a chance to reset and start over with another topic. Ask and move on – don’t get stuck on getting them to reach the right answer.

V – Decision

That’s it. We meet as a team and get everyone’s feedback. We should have an answer the following day.

Final Thoughts

Like most people, I don’t like going to interviews where I don’t know the team and I am asked to “perform”. The interview should adapt to your level. Architects should not be asked to solve short algorithms, the same way managers should not be asked to write code. Your references and past work is what we should really rely on for verifying your claims. The interview is mostly about personality, compatibility, and due diligence.

A job interview goes both ways. We interview you, and you interview us. We want great talent and respect is the first step.

hapi hapi joi joi

2013-01-18T01:42:12Z

Download the slides as a PDF document.

hapi, a Prologue

2012-12-20T08:33:44Z

For the past 2 years I have had the pleasure (and luck) of working with node full time. It’s an amazing technology and a remarkable community. Oh, and it’s crazy fun. My focus this year was rethinking web services at Walmart Mobile, from the business layer all the way down to the tools and process we use. A significant part of this effort focused on developing hapi, a new web services framework for node.

But before I write my traditional ‘Introducing’ post, I wanted to first discuss the evolution that led us to build a whole new framework. To truly understand the judge a new framework, it is important to understand the context and objectives leading to its creation.

Walmart, like many big and small companies, is not in the business of creating and maintaining developer tools.

There is great value in doing open source and participating in the community. But without an active community, developing an open source project can quickly become a liability. When we started, the goal was to take an existing effort and simply enhance it to meet our goals. We didn’t set to build an entire stack from scratch. If you build it and nobody comes, it can get expensive and very isolating.

I’ve started using node working on Sled, a short-lived project at Yahoo! which was open-sourced as Postmile. Sled had a simple services architecture in which multiple clients communicated with an API server. At the time, the only real framework available was Express, and for a very early effort, it was quite good.

But it also required a lot of effort supplementing it with the necessary building blocks for a productive foundation. Most of the effort focused on changing the interaction between Express and the business logic, along with support for authentication, encrypted cookies, input validation, and other small features missing at the time.

When Sled ended and I moved on to Walmart, I started by extracting the parts of Postmile I liked as the foundation of our new services platform and called it hapi (for HTTP API server). It was still built on top of Express, and wasn’t much more than a bloated Express middleware – but it was easy to use and much faster to hack services on. It was a good start.

We used Express for a few months until we hit a wall.

Some of the features we needed required a cleaner isolation between the node HTTP server, the router (the part matching requests to handlers), and the actual business logic. Express tries to be an unopinionated framework, but in practice, there is no such thing. Every decision made imposes an opinion and a way of working with the framework, and Express comes with a lot of baggage from its Connect roots (req, res, next).

For example, we were trying to add batch functionality in which a JSON payload includes a list of chained endpoints, some parallel and some serial. Given the cost of round-trips in mobile, this was an important requirement. The problem was, because of Express’ middleware architecture and it’s strong coupling of the router to the node HTTP server, the only clean way of making internal API calls that truly replicate the result of calling each API individually from the outside is to make an actual HTTP call into the server. This gets real messy when using authentication and having to fake client credentials.

We also had some bad experience with Express’ lack of true extensibility model. Express was a pleasure and easy to use 2 years ago with a limited set of middleware and very little interdependencies among them. But with a long list of chained middleware, we found hard to debug problems when we simply changed the order in which middleware modules were being loaded.

Goodbye Express, hello Director, the simple but powerful router from Nodejitsu’s Flatiron project.

Replacing Express with Director took a day and the result was a cleaner interface between the layers. We enjoyed working with Director and the responsive support from its active community. By this time, there was little resemblance between our higher level framework API and Express’ since we wrapper the entire Express API with our own. We were able to quickly implement batch processing and other features and used Director successfully.

But as you can guess, that didn’t last long.

In order to achieve Director’s highly optimized routing tree, it had to make certain choices about how paths are structured and configured. Director offers a particular mix of static routes, parameterized routes, and some forms of wildcards / regular expressions.

In some ways, Director’s attempt to accommodation expectations from Express users ends up make it more complex for little gain. Director does not keep track of the path parameter names (only their order) which meant we had to hack that feature back in. We also had to work around Director’s way of providing extensibility hooks via before/after functions. After all, how framework are extended is one of the most significant architectural choices.

One of the main problems we had with Director was the difficulty in prevent route collisions – a critical feature when building a modular web services layer where different plugins register their endpoints with a central host. More on that in a followup post.

We also looked at Restify, Tako, and a few others.

We didn’t set to create a whole new framework, but at the end of the day, it was easier and more appropriate than trying to make an existing solution work. Framework represent a philosophy encompassing both coding style and engineering process, and it’s hard to impose one on another.

hapi was created around the idea that configuration is better than code, that business logic must be isolated from the transport layer, and that native node constructs like buffers and stream should be supported as first class objects. But most importantly, it was created to provide a modern, comprehensive environment in which as much of the effort is spent delivering business value.

A formal hapi introduction is coming. If you can’t wait, check it out.

#fuckoauth @realtimeconf

2012-11-05T23:44:50Z

On Leaving OAuth

2012-07-30T09:11:31Z

I have to admit, I’ve been surprised by the tone and magnitude of the reaction to my announcement. I’ve been following the reactions on Twitter and every follow-up blog post I could find. I’d like to share some follow-up thoughts about this decision and the reactions to it. First, I didn’t say ‘dead’, I said ‘bad’.

Everything is fine

Few people came to the defense of the protocol, the working group, and the IETF. Surprisingly few. In fact, no one has raised the issue on the mailing list. This is not exactly a Hy-Brazil moment, but some of the reactions do make me wonder.

One response worth reading was from John Bradley in a post titled “The OAuth 2 Sky is NOT Falling”. John is a past collaborator (XRD) and someone I respect and his post offers a quality reaction from someone on the other side of the issue. As a leading contributor and co-author of the OpenID Connect family of specifications, it is easy to understand why John and that community are happy with OAuth 2.0 and the process it followed.

OpenID Connect is heavily based on OAuth 2.0 – I personally find it overly complex. Most of the OpenID Connect members joined the OAuth list in order to protest decisions that would have made it too restrictive for OpenID Connect to use OAuth 2.0, and their participation explains a few of the underspecified areas of the specification.

Another community that has been very satisfied with OAuth 2.0 is UMA. Some of the UMA project leads are people are like and respect, like Eve Maler. In the past I have invited members of the UMA community to share their project with the OAuth community on the mailing list, at IETF meetings, and on this blog. It has been a long time since I read up on UMA but I was always skeptical about its relevancy to the consumer web world I care about. UMA is also based on OAuth 2.0 and relies on many of its extensibility areas to operate. If you want to get an idea of the complexity (and richness) of this world, this is a good place to start.

One last example of an OAuth 2.0-based work that can help explain and justify the “decisions made” on OAuth 2.0 is OMAP (Online Multimedia Authorization Protocol). OMAP is an enterprise specification dealing with authorizing access to multimedia content and was authored by lead players in the space. I honestly don’t understand the value of basing this work on OAuth 2.0 or how such an extension could be claimed to operate in the same echo-system. It is the perfect example of the real use cases guiding the working group and the real-world application some members of the group were focusing on.

So, yeah. For some people “the sky is not falling” and OAuth 2.0 is a success. It is easy to see where they are coming from and why they hold this view. If OAuth 2.0 was a contest, they are clearly the winners and they won fair and square by playing by the IETF rules. I just don’t want to lend it my name, whatever it is worth.

Eran is an asshole

I fully respect the opinions of those who have directly interacted with me in person or online and formed positive or negative opinions about me. My style is aggressive, direct, honest, and often combative. I don’t (and cannot) pretend otherwise. All it takes is reading a small sample of my blog posts or mailing list communications to get a clear and accurate sense of who I am and what I’m like. I am exactly the same way in person.

But this also means that three years into this effort, to paint my move in negative personal attacks is childish and hypocritical, especially from people who never interacted with me directly (or those who pretended to be my friends).

In a comment to my announcement, Dick Hardt in a personal attack, points to the fact that I insisted on full editorial control and therefore, can only blame myself. The facts are somewhat different. I was offered to join the editorial team with two others. I declined and said I don’t edit with other people, but happy to let them do the work without me. I was made solo editor. It was all very pleasant. The IETF process makes it trivial for the working group chairs and area director to remove, replace, or supplement an editor at will. They really don’t need any reason.

A few months ago, when I started considering removing my name from the specification, I reach out to Mr. Hardt as a listed co-author on the document. His advice? “I don’t think it will serve you well. Better to stay the course and be disappointed with result. You can remain editor and blame IETF post editor role process.”

Facebook and Google are doing just fine thank you

A quality, respectful, and worthy response from Tim Bray makes the argument that clearly, the existing (and very successful) deployment of OAuth 2.0 on the web tells a different story. I don’t disagree. As I said before, in the right hands, after applying the right security and making some decisions, OAuth 2.0 can be great. Facebook made those decision (one of which was to ignore the last 20-something drafts) and so did Google. However, even Facebook with their vast resources had issues with 2.0 and ended addressing them with proprietary solutions (e.g. a callback signature, login parameters).

I would have been thrilled if the final result was a documentation of the Google or Facebook implementations and nothing else. That would have been a useful protocol.

If you are not sure what I am so upset about, it is the total lack of making choices. Given the new architecture and capabilities, OAuth 2.0 should have been much more restrictive and narrow.

The IETF is good, the IETF is bad

A few reactions focused on the IETF part of the story. No one seemed to offer an actual defense of the organization. Joe Gregorio offered one only to immediately take it away. At best, people pointed to past successes at the IETF as proof the organization is still relevant and effective. The problem is that their examples are all from two decades ago. The second line of defense simply states that standards are hard and that’s just the nature of the beast. OAuth 1.0 proves it doesn’t have to be.

Glee

I can live with all the reaction above. Even those dismissing my arguments or calling me an asshole. I expected those. What I didn’t expect was an overwhelming expression of joy and schadenfreude. I didn’t realize just how much this specification was hated among web developers. I knew people were not thrilled about it but never imagined the level of discontent.

What is actually upsetting is that all these people (and we are talking hundreds of individual posts on Twitter) never bothered to engage. Three years is a long time to hate something this much without sending an email to the mailing list expressing it. If only a fraction of these people showed up, we would not be here today. People actually were paying attention but doing nothing about it.

Now, that’s just sad and pathetic.

(As always, comments are welcomed)

OAuth 2.0 and the Road to Hell

2012-07-30T09:03:21Z

They say the road to hell is paved with good intentions. Well, that’s OAuth 2.0.

Last month I reached the painful conclusion that I can no longer be associated with the OAuth 2.0 standard. I resigned my role as lead author and editor, withdraw my name from the specification, and left the working group. Removing my name from a document I have painstakingly labored over for three years and over two dozen drafts was not easy. Deciding to move on from an effort I have led for over five years was agonizing.

There wasn’t a single problem or incident I can point to in order to explain such an extreme move. This is a case of death by a thousand cuts, and as the work was winding down, I’ve found myself reflecting more and more on what we actually accomplished. At the end, I reached the conclusion that OAuth 2.0 is a bad protocol. WS-* bad. It is bad enough that I no longer want to be associated with it. It is the biggest professional disappointment of my career.

All the hard fought compromises on the mailing list, in meetings, in special design committees, and in back channels resulted in a specification that fails to deliver its two main goals – security and interoperability. In fact, one of the compromises was to rename it from a protocol to a framework, and another to add a disclaimer that warns that the specification is unlike to produce interoperable implementations.

When compared with OAuth 1.0, the 2.0 specification is more complex, less interoperable, less useful, more incomplete, and most importantly, less secure.

To be clear, OAuth 2.0 at the hand of a developer with deep understanding of web security will likely result is a secure implementation. However, at the hands of most developers – as has been the experience from the past two years – 2.0 is likely to produce insecure implementations.

How did we get here?

At the core of the problem is the strong and unbridgeable conflict between the web and the enterprise worlds. The OAuth working group at the IETF started with strong web presence. But as the work dragged on (and on) past its first year, those web folks left along with every member of the original 1.0 community. The group that was left was largely all enterprise… and me.

The web community was looking for a protocol very much in-line with 1.0, with small improvement in areas that proved lacking: simplifying signature, adding a light identity layer, addressing native applications, adding more flows to accommodate new client types, and improving security. The enterprise community was looking for a framework they can use with minimal changes to their existing systems, and for some, a new source of revenues through customization. To understand the depth of the divide – in an early meeting the web folks wanted a flow optimized for in-browser clients while the enterprise folks wanted a flow using SAML assertions.

The resulting specification is a designed-by-committee patchwork of compromises that serves mostly the enterprise. To be accurate, it doesn’t actually give the enterprise all of what they asked for directly, but it does provide for practically unlimited extensibility. It is this extensibility and required flexibility that destroyed the protocol. With very little effort, pretty much anything can be called OAuth 2.0 compliant.

Under the Hood

To understand the issues in 2.0, you need to understand the core architectural changes from 1.0:

Unbounded tokens - In 1.0, the client has to present two sets of credentials on each protected resource request, the token credentials and the client credentials. In 2.0, the client credentials are no longer used. This means that tokens are no longer bound to any particular client type or instance. This has introduced limits on the usefulness of access tokens as a form of authentication and increased the likelihood of security issues.
Bearer tokens - 2.0 got rid of all signatures and cryptography at the protocol level. Instead it relies solely on TLS. This means that 2.0 tokens are inherently less secure as specified. Any improvement in token security requires additional specifications and as the current proposals demonstrate, the group is solely focused on enterprise use cases.
Expiring tokens - 2.0 tokens can expire and must be refreshed. This is the most significant change for client developers from 1.0 as they now need to implement token state management. The reason for token expiration is to accommodate self-encoded tokens – encrypted tokens which can be authenticated by the server without a database look-up. Because such tokens are self-encoded, they cannot be revoked and therefore must be short-lived to reduce their exposure. Whatever is gained from the removal of the signature is lost twice in the introduction of the token state management requirement.
Grant types - In 2.0, authorization grants are exchanged for access tokens. Grant is an abstract concept representing the end-user approval. It can be a code received after the user clicks ‘Approve’ on an access request, or the user’s actual username and password. The original idea behind grants was to enable multiple flows. 1.0 provides a single flow which aims to accommodate multiple client types. 2.0 adds significant amount of specialization for different client type.

Indecision Making

These changes are all manageable if put together in a well-defined protocol. But as has been the nature of this working group, no issue is too small to get stuck on or leave open for each implementation to decide. Here is a very short sample of the working group’s inability to agree:

No required token type
No agreement on the goals of an HMAC-enabled token type
No requirement to implement token expiration
No guidance on token string size, or any value for that matter
No strict requirement for registration
Loose client type definition
Lack of clear client security properties
No required grant types
No guidance on the suitability or applicability of grant types
No useful support for native applications (but lots of lip service)
No required client authentication method
No limits on extensions

On the other hand, 2.0 defines 4 new registries for extensions, along with additional extension points via URIs. The result is a flood of proposed extensions. But the real issues is that the working group could not define the real security properties of the protocol. This is clearly reflected in the security consideration section which is largely an exercise of hand waving. It is barely useful to security experts as a bullet point of things to pay attention to.

In fact, the working group has also produced a 70 pages document describing the 2.0 threat model which does attempt to provide additional information but suffers from the same fundamental problem: there isn’t an actual protocol to analyze.

Reality

In the real world, Facebook is still running on draft 12 from a year and a half ago, with absolutely no reason to update their implementation. After all, an updated 2.0 client written to work with Facebook’s implementation is unlikely to be useful with any other provider and vice-versa. OAuth 2.0 offers little to none code re-usability.

What 2.0 offers is a blueprint for an authorization protocol. As defined, it is largely useless and must be profiles into a working solution – and that is the enterprise way. The WS-* way. 2.0 provides a whole new frontier to sell consulting services and integration solutions.

The web does not need yet another security framework. It needs simple, well-defined, and narrowly suited protocols that will lead to improved security and increased interoperability. OAuth 2.0 fails to accomplish anything meaningful over the protocol it seeks to replace.

To Upgrade or Not to Upgrade

Over the past few months, many asked me if they should upgrade to 2.0 or which version of the protocol I recommend they implement. I don’t have a simple answer.

If you are currently using 1.0 successfully, ignore 2.0. It offers no real value over 1.0 (I’m guessing your client developers have already figured out 1.0 signatures by now).

If you are new to this space, and consider yourself a security expert, use 2.0 after careful examination of its features. If you are not an expert, either use 1.0 or copy the 2.0 implementation of a provider you trust to get it right (Facebook’s API documents are a good place to start). 2.0 is better for large scale, but if you are running a major operation, you probably have some security experts on site to figure it all out for you.

Now What?

I’m hoping someone will take 2.0 and produce a 10 page profile that’s useful for the vast majority of web providers, ignoring the enterprise. A 2.1 that’s really 1.5. But that’s not going to happen at the IETF. That community is all about enterprise use cases and if you look at their other efforts like OpenID Connect (which too was a super simple proposal turned into almost a dozen complex specifications), they are not capable of simple.

I think the OAuth brand is in decline. This framework will live for a while, and given the lack of alternatives, it will gain widespread adoption. But we are also likely to see major security failures in the next couple of years and the slow but steady devaluation of the brand. It will be another hated protocol you are stuck with.

At the same time, I am expecting multiple new communities to come up with something else that is more in the spirit of 1.0 than 2.0, and where one use case is covered extremely well. OAuth 1.0 was all about small web startups looking to solve a well-defined problem they needed to solve fast. I honestly don’t know what use cases OAuth 2.0 is trying to solve any more.

Final Note

This is a sad conclusion to a once promising community. OAuth was the poster child of small, quick, and useful standards, produced outside standards bodies without all the process and legal overhead.

Our standards making process is broken beyond repair. This outcome is the direct result of the nature of the IETF, and the particular personalities overseeing this work. To be clear, these are not bad or incompetent individuals. On the contrary – they are all very capable, bright, and otherwise pleasant. But most of them show up to serve their corporate overlords, and it’s practically impossible for the rest of us to compete.

Bringing OAuth to the IETF was a huge mistake. Not that the alternative (WRAP) would have been a better outcome, but at least it would have taken three less years to figure that out. I stuck around as long as I could stand it, to fight for what I thought was best for the web. I had nothing personally to gain from the decisions being made. At the end, one voice in opposition can slow things down, but can’t make a difference.

I failed.

We failed.

Some more thoughts…

You, Me, and Node @WalmartLabs

2012-01-04T23:04:20Z

Another adventure begins.

Two months ago I’ve joined @WalmartLabs to lead the mobile web services team. Surprised? I was. After working for one of the largest web companies in the world, all I wanted to do was go to a startup. That’s not exactly right; I wanted to be part of a tiny team with a big mission, a place where the size of the challenge is matched by the freedom and resources to address it. Oh, and a lot of node.js!

I am excited to share this and tell you all about it, especially on the heels of this morning announcement of the acquisition of Small Society. But I’m not going to lie to you: I have an agenda and I am trying to recruit you. If you are contemplating a career move and I “had you at node”, feel free to jump right to very end of this post to find more about the team we’re building.

Rethinking Retail Through Mobile

It’s all about size, especially when it comes to consumer retail. @WalmartLabs is part of Walmart’s Global E-Commerce group and is focused on reinventing retail through technology. This is among the most important strategic investments Walmart has made in recent years and it goes deep and wide. This is not simply about making online shopping better, but about fundamentally changing retail by creating a new, continuous shopping experience across technologies and venues.

Our team is focused specifically on executing this vision through mobile. Unlike most other mobile retail applications, our focus isn’t limited to buying stuff on your phone or tablet. We are looking at the full range of possibilities when you have a powerful mobile device in your hand, both inside and outside the store. In addition, we are looking at a much wider set of products, from electronics to pharmacy and all the way to fresh groceries.

The best part about building a team and talking to potential new hires are the new product ideas that come up when you start to think about what’s possible: An app that sorts your shopping list based on aisle location as you walk into a store; the brick-and-mortar version of ‘60% of your friends bought this brand of toothpaste’ when trying to decide which to buy; real-time checkout, showing you the total dollar amount including tax of the items in your cart; shopping analytics across stores and online purchases producing a new and unmatched dataset about your lifestyle and health as the foundation of new tools to improve your life. And this is just from my most recent conversation.

From my first meeting with the team, I found myself thinking about new ways to change retail. I’ve never given much thought to that, other than noticing a feature I liked on Amazon or a time-saving service at a store. All of a sudden, I’ve started noticing all the small things we can change today – the obvious ideas just lying on the floor – if only I could get those apps in the hand of enough people. And that’s where Walmart offers a unique opportunity, with hundreds of millions of loyal customers worldwide eager to give your new idea a try.

Working for Walmart? You Can’t Be Serious!

Now let’s talk about the big blue elephant in the room: working for Walmart. I’m in no way trying to change how you view Walmart; only to share with you the process I went through.

To put it bluntly, I had a gag reflex reaction to the idea of working for Walmart. It’s hard for me to pinpoint exactly what my reaction was based on. Walmart’s image, history, and impact on society played a role, but it was mostly my recent experience working for a large web company and past experience working for Citibank, the largest bank in the world, that turned me off. Short of working for the Department of Defense or the Chinese Army, nothing gets bigger than Walmart.

This was not the employer of my dreams – far from it! – but the opportunity itself, and more importantly the caliber of the team, convinced me to take a deeper look. I spent hours reading up on Walmart, the good and the bad. Given the size of the company, there was plenty of both. I was surprised at how much I didn’t know about the company.

I was mostly blown away by the numbers, of which two really stuck with me:

in the United States, 140 million people visit a Walmart store every week, and
Walmart is responsible for 35% of all groceries sold.

I remembered an article I read a few years ago about Adam Werbach, once the youngest president of the Sierra Club, and his decision to join Walmart and the backstory of that decision. I also recalled the Stonyfield story in Food Inc. and the impact Walmart had on the landscape of organic food (price, availability, and commoditization) once they decided to carry it in their stores.

I’m not trying to draw any comparisons between these stories and my personal decision, other than to point out that when you get to point the “Force” (or “Death Star”, depending on your perspective) at a target, big things happen.

At the end, we all make compromises when we work for any corporation, especially those large enough to have real impact. Walmart’s size translates directly into a noticeable impact on society but without getting too philosophical, so does Google’s, Apple’s, and News Corp’s to name a few. I have major grievances with all these corporations, but I do recognize that if I was trying to recruit you to work there, I would not be spending this much time to get you to look past the brand’s reputation. I respect the fact that for some it will always be “you lost me at Walmart”.

What Do I Want to Do When I Grow Up?

Last year I wrote about my decision to stay at Yahoo! and work on Sled, a virtual startup within Yahoo! building a collaborative list-making tool. The list included products, culture, manager, team, and work/life balance, and these are still the main factors that matter to me. When the time came to move on to another opportunity, I’ve asked myself a two questions: what is it I’m most excited about these days, and who is it I am most interested in working with?

My passion and focus the past four years has been around APIs: frameworks, design, advocacy, and lot of security (you know, all that OAuth stuff). But my true joy over that period is a lot more specific: node.js. Working with node for over a year in a full-time capacity has spoiled me and for the first time in a while, added a specific technology to my job search criteria.

In one sentence, my dream job was leading the development of a new, world-class, open-sourced, node-based, OAuth-protected, developer-focused, API server framework.

I wanted to take all my experience and ideas from the past few years and apply them to one awesome project. But that wasn’t enough (is it ever?). It had to be real. I didn’t want another academic exercise or working in a vacuum. I also had no energy to constantly fight my employer to let me do what I was hired for. I wanted to build something that will get used and abused as soon as it was forming, ideally by millions of users.

I wasn’t expecting to find it but I actually ended up with two such opportunities – both amazing. One was an early stage startup and the other, Walmart. The Walmart opportunity was a surprise. One of the people I really wanted to work with was Dion Almaer. We have not worked together before but we did cross paths over the years and I’ve been following Dion’s and Ben Galbraith’s work with interest. Turned out that earlier this year they sold their company to Walmart and took over mobile engineering, helping to create @WalmartLabs.

Ben and Dion’s story on how they got to Walmart is pretty interesting by itself but even more telling about the team and the kind of opportunities it offers. At the end it wasn’t an easy decision and it was very close. I’m happy to report that a 2 months into this, I’m super excited about my decision and it seems to be the prevailing attitude shared by the rest of the brilliant people I get to work with which include Kevin Decker leading the mobile web team, Stefan Li leading the Android team, Brandon Sneed leading the iOS team, and the new folks from Small Society to mention just a few.

Let’s Talk About You!

Now that you know how I ended up at Walmart and got the big picture of what we are trying to do, we can talk about what it is we are actually doing and how you might fit in.

To expand a bit on my one sentence job description, the mobile services teams is in charge of the backend serving all of our mobile applications. Today these include iPhone, iPad, Android, and mobile web. We are mainly focused on the US market but will expand internationally in the coming months. We have a legacy JAVA system in place which we are going to replace over the next few months with a brand new node solution, as well as develop new services directly in node.

Our main focus is building a world-class web services operation which includes building the best API server framework possible. Think of it as Express for APIs, with the typical requirements for speed and flexibility, but designed in a way that enables building an entire API service in days and focus primarily on business logic instead of facilities (e.g. query and payload validation, smart routing, dynamic documentation, OAuth support). And of course, we are going to build the core of this framework as open source from the very beginning.

In addition, we are looking to build and contribute to projects aimed at making mobile services better, including smart caching (we are currently evaluating Redis and other new technologies as the foundation of such facilities), and dynamic repositioning of HTML rendering services between the back-end and front-end. Also, in alignment with our ‘Labs’ name, we experiment with new technologies and architectures aimed at improving mobile performance end-to-end.

The current services team is about four people and we are looking to add a few more over the next few weeks. The new roles will be 100% node, working on the new framework and services. Unlike most other “node jobs”, this is really all about node, not a trick to get smart people excited about yet another JavaScript gig.

If you noticed, I haven’t wasted your time with stories about how, despite the size of the company, it still “feels just like a startup”. After all, this is what every large company tries to sell these days. Instead, I invite you to come and check us out, talk to the team and form your own opinion on how lean and mean we really are. Most of our team joined us from recent startup experiences and they care a lot about that culture and energy. Our team is fairly distributed with the main office located in a new building in San Bruno, CA. The iOS team is based in Portland, OR, and we have quite a few team members working remotely.

We are looking for brilliant engineers passionate about new technologies in general and node.js in particular. We are not going to bore you with a useless list of experience and requirements. If you think you have what it takes to get the job done, we want to talk. We promise not to bring you in for an endless parade of grilling interviews or ask you to write code on a whiteboard. We much rather check out your Github account and have a lively debate about technologies, software design, and your obligatory bizarre geek hobbies (we’ve got our own to share).

I will be writing about our progress and the new framework over the next few months. This promises to be an exciting ride. If you want to learn more, drop me a line.

Sled, Yahoo!, and Moving On

2012-01-04T23:36:13Z

As is often the case when working on a startup, virtual or not, things don’t work out according to plan. I had the privilege to spend the past year working on Sled, from product inception to execution. At the same time we were launching Sled, I experienced some significant management changes at Yahoo! which eventually led to shutting it down.

To be clear, the decision to discontinue Sled was mine alone, but it wasn’t made in a vacuum. The change in management brought with it a change in attitude towards the project in both form and substance. My options were to stay and fight for it, or shut it down and move on. It was pretty upsetting, as these events usually are, especially given how active the list-making / light collaboration space is and the huge potential there. After fighting the organization for three years, I decided to move on and left the company shortly after.

This was the official announcement posted at the time on the now defunct Sled blog:

Over the past few months, I had the pleasure of being part of an exciting experiment at Yahoo! called Sled.

Sled was a collaborative list making tool with a strong focus on life events and collaboration between friends and family. Planning a party, a house move, getting ready for a new baby, planning a trip, organizing a junior soccer league, or preparing for a marathon, are some of the areas Sled was focused on. We also found it really useful for building Sled itself, working in a small team, keeping track of issues and assignments.

There is a lot we don’t know about how to make our daily lives more productive and organized, and how to collaborate better. What we do know is that the wide range of tools and services available to us are, generally speaking, not very helpful. Everything is either too limited or too complicated. Usually too complicated. We know what doesn’t work, but figuring out what does requires experimentation.

Sled was developed as a virtual startup at Yahoo!, using open technologies and the freedom to experiment. As is often the case with startups and experiments, we have been constantly evaluating the product and its fit within our existing and future roadmap, and have made the decision to discontinue the project in its current form (shutting down sled.com at the end of the week).

But our story doesn’t end there.

We built Sled on an entirely open stack, using open source tools (Node.js, MongoDB, Express, Socket.IO, Jade) and open standards (JS, HTML5, OAuth 2.0). We have relied and benefit from a vibrant community of amazing developers and we want to give something back. One weak area for the community is the availability of fully-baked, showcase applications written in Node.js.

Instead of following the typical industry path of discontinued products, we have decided in the experimental spirit of Sled to release the entire project under an open source license using a new name: Postmile. This means anyone will be able to grab the code and run the entire service, back and front, exactly as it was hosted on sled.com (including the soon to be open sourced iPhone app we never got to release).

We hope you will find Postmile helpful, fun, and an insightful resource for the kind of projects you can build with Node.js today. Check it out at http://j.mp/postmile. You can use Postmile as a back-end API server for building new list-based products (to-do apps, simple project management tools, bug tracking), as a list making tool for your friends or your company, or just as a useful example to grab code from.

The best part about working on Sled was node. I got the rare opportunity to spend a year working full time on node and be part of the node community. The experience was so fantastic that I made working with node a central theme of my job search and I’m happy to report that it worked out quite nicely. But that’s for another post.

Is the Party Winding Down at Facebook?

2011-12-12T17:03:33Z

A picture started to emerge from casual conversations I’ve had over the past few weeks with friends working at Facebook. I have noticed how Facebook engineers are using a different, more restrained vocabulary to describe their jobs. What once was ‘amazing’ is now ‘challenging’, ‘exciting technology’ turned to ‘learning a lot’, and ‘having fun’ toned down to ‘still engaged’. They are all very ‘content’.

This might be purely anecdotal but I have a feeling it is not. I’ve shared this insight with a few other friends outside of Facebook and they too started noticing the pattern within their own circle of Facebook connections. Unlike a year ago, these days I can’t point to a single person I know working at Facebook who is oozing excitement and energy. They are still very much engaged and are busy as ever, but they are less interested in talking about work.

While this kind of corporate evolution is expected, it is a bit early for Facebook to make the transition from a hyper startup to a business-as-usual large company culture. If this continues, the looming IPO is going to include a noticeable loss of talent with what feels like a new holding pattern within the company of people waiting to fully vest or cash out.

Netflix Forcing the Issue Too Soon

2011-12-13T04:22:30Z

(A note to my long-time readers, I’m planning on expanding this blog to include opinions about current technology trends and news beyond my usual fare of standards, open web, and engineering posts.)

This morning I logged into my Netflix account and changed my plan from 3 DVDs + Streaming to DVDs Only. Despite the excellent analysis by many about Netflix’s reasons for the recent changes, the fact is that today the company is making $95.88 less annually from me than they did the day before. That’s a lot of money to lose from a single, loyal customer.

I love the unlimited streaming service, but given its current selection of movies, I use it mostly to catch up with TV shows I missed or skipped. We do this mostly during the summer TV hiatus. In other words, the current streaming catalog is great when there is nothing else to watch and you are willing to compromise with lesser, somewhat stale, content.

I’m probably going to get a streaming service back in June when the 2012 season is over. But at that point I will no longer default to Netflix because I have no incentive to do so. It doesn’t save me money or add any nice features to my core DVD-by-mail service (today, I can see what in my queue is available on-demand and consume it that way). I will definitely consider Netflix as an option, but if my cable provider, Apple TV, or XBOX, offer me an easier alternative with the right content, I’m probably going to take the lazy way out.

Strategically, this was the right move. Tactically, it’s a big risk to take given the blow it can serve to their momentum. Everything communicated by the company makes perfect sense to me. The problem is that by pushing the issue now, they have forced me to think about their separate services when their weaker offering is the one they are betting their future on.

The Unauthorized Node Knockout #2 Awards

2011-09-04T07:32:09Z

I’ve spend the past week participating as a judge in the 2nd Node Knockout competition – a 48 hours worldwide hackathon using node.js. The event included 720 contestants organized in 294 teams and resulted in 178 entries submitted for review. Overall, a fantastic event and a testament to the awesomeness that is the node community.

The competition includes prizes for the best hack in the following categories: fun or utility, design, innovation, completeness, popularity, and overall, as well as overall for a solo participant. While judging is still on-going, and while many of the top entries do deserve to be there, I found the categories to be a bit uninspiring.

Also, as the only crazy person to judge every single entry (including some that dropped out in the process), I wanted to highlight some entries that might not get the attention they deserve. The following are the nominees and winners in my unauthorized awards.

Most Geek-elicious

And the winner is: Node Defense

Biggest Time Suck

And the winner is: Metris

Most Practical

And the winner is: nide

Best Art Direction

And the winner is: Gemini Command

Most Original Classic

And the winner is: Pong

Most Bizzare

And the winner is: Boys over Flowers

Biggest WOW Moment

And the winner is: LocalNode

Congratulations to everyone who participated. I highly recommend you to check out every single entry listed on this page. This is the best of the best – and I’ve seen them all! Also, a big shout out to Gerad Suyderhoud and Visnu Pitiyanuvath for the amazing job they have done organizing the event.

Can’t wait for next year!

Twitter Accounts

2011-08-02T02:22:22Z

Just a quick update. I’ve split my twitter account into two. Follow @hueniverse for blog updates and other information about the technical topics I discuss on this blog. Follow @eranhammer for my personal brain farts.

A Farmer Walks into Facebook

2011-07-19T06:57:15Z

A few weeks ago I went to visit some friends at Facebook’s headquarter. We had an interesting chat about OAuth and other geek topics. As is common these days, the conversation drifted to my recent adventures in farming. I was describing my setup, the chickens, ducks, geese, and pigs I’ve got running around, and then mentioned my three emus all named Kevin.

Someone who was listening in from a nearby cube stood up and asked, “When did they add emus?”

Get it? At Facebook everyone’s a farmer.

OAuth 1.0 Blog Cleanup

2011-07-16T06:33:04Z

As I’m getting ready to finish work on OAuth 2.0 and add new content to this site, I decided it was time to finish the OAuth 1.0 chapter of this site. I’ve finally cleaned up the OAuth 1.0 guide and other pages. The guide is now updated to reflect RFC 5849 as well as some bug fixes in the scripts used to generate the signature base string tutorial. If you are linking to this site for OAuth resources, please link to the OAuth page.

Introducing Sled

2011-07-14T22:17:55Z

I’ve been obsessed with project management and personal productivity for a almost two decades. My experience ranges from tiny lists to gigantic project plans with hundreds of people and resources. In the past I’ve been a certified PMP and managed large engineering teams. What I’ve learned above all, is that we tend to overcomplicate everything.

Four years ago my startup Nouncer failed for many reasons, none of which had much to do with the product itself. Looking at where Twitter is now and how it evolved, it is a clear validation of my original vision. But even if I had gotten passed the challenges that doomed Nouncer, I still think it would have failed. It was just too complicated, too soon.

I’ve long considered Twitter’s biggest asset to be its 140 character limit. It completely democratized personal expression by making everyone as expressive and articulate. It also helped people communicate more by making their content small enough for casual, constant consumption.

A year ago I started thinking about applying this philosophy – empowerment through restrictions – to project management. I’ve started thinking about enterprise-scale problems and what a restrictive tool might look like. But no longer working on large scale enterprise projects, my attention shifted to personal productivity and “home projects”, and so Sled was born.

Sled is an experiment.

There is a lot we don’t know about how to make our daily lives more productive and organized, and how to collaborate better. What we do know is that the wide range of tools and services available to us are, generally speaking, not very helpful. Everything is either too limited or too complicated. Usually too complicated. We know what doesn’t work, but figuring out what does requires experimentation.

Sled is a collaborative list making tool with a strong focus on life events and collaboration between friends and family. Planning a party, a move, getting ready for a new baby, planning a trip, organizing a junior soccer league, or preparing for a marathon, are some of the efforts I hope Sled will be useful for.

The idea is to start with a really solid tool for making lists. Then add collaboration with a few people, a little bit of social features, and later a crowd-sourced knowledge base of lists and recommendations. For the past few months we’ve been focusing on list making and the result is a simple tool to create lists alone or with a small group of friends. Sled is still in its very early stages, but it is usable and (we hope) fun.

I’m not planning on writing much about Sled on this blog. Instead, I will focus on sharing my engineering experience building the product. If you want to keep up to date with Sled, follow us on Twitter or Facebook, or check out the Sled blog.

And if you want to check it out, sign-in with your Yahoo!, Twitter, or Facebook account and use invite code: hueniverse.

Node.js: Express, Socket.io, and everything LearnBoost

2011-07-14T22:20:56Z

This post is part of a series of articles about my recent experience building Sled using Node.js.

Express It

There wasn’t much of selection 6 months ago when I started coding Sled when it came to Node frameworks. Node itself provides very little. You create an HTTP server and get a callback when a new HTTP request comes in. Modern web applications require session management, request routing, and view rendering at the very least. These functions are provided by frameworks.

At the time, the only two popular frameworks were Connect and Express. Connect provides a basic middleware framework with some built-in facilities such as a static file server, route management, cookie parser, form-encoded and JSON-encoded body parser, and logging. Express, built on top of Connect, adds a much more robust routing facilities, view rendering and other goodies. I’m not doing these frameworks justice with this descriptions.

I use Express extensively and it is fantastic.

Express’ biggest asset is its maintainer, TJ Holowaychuk. I have seen first-hand how Express grew and matured into a stable and powerful framework. Express made it easy for me to write the application server used for login, registration, account management, and other static assert (developer pages, about) with very little extra effort. If you are going to develop a traditional web application in Node, you should probably start with Express.

I plan to open source some of the additional functionality I’ve added on top of Express to validate request bodies and query parameters, a flexible authentication configuration facility for routes, and a light layer to make it easier to building API servers. Given my focus on OAuth, I’m going to share my OAuth + Express experience in a followup post.

Express really shines when you combine it with Jade, another brilliant brainchild of Mr. Holowaychuk – a simple templating language for HTML which is easy to learn, easy to read, and unlike all the rest, doesn’t suck. We had to restrain ourselves from converting some static HTML files into Jade because once we started using it, we didn’t want to read actual HTML ever again.

Socket.io is Magic

Really!

I am willing to bet that a large percent of those taking a look at Node for the first time are doing it because of a Socket.io -powered demo they have seen. Socket.io provides a trivial-to-use server and client library for making real-time, streaming updates between a web server and browser client. It makes building cool real-time games a matter of hours, and it works on every crappy browser known using the best available option from Flash to WebSockets.

We use Socket.io to power Sled’s real-time features which include live updates to your shared list as changes are made. To put this in perspective, it took us one day to add real-time streaming updates to the API server, and another day or two to add it to the client. So in three days we got full, real-time updates going between multiple browsers. What used to be months of work when Google Docs was first introduced, is now trivial with Socket.io.

So yeah, it’s magic.

Socket.io comes from Guillermo Rauch and the team of superstars at LearnBoost. There are days when I have to wonder if these guys get anything done for their own startup, given the amazing open source projects they push out on a weekly basis. I’m bummed that I’m not the target audience for their product because looking at the technology it must be amazing. I consider them part of the Sled team, given just how much of their code we are using.

In fact, I appreciate their work so much, I’d like to get a small fundraiser going to send these guys to dinner or whatever else they feel like doing for fun. I hope you join me in showing our appreciation. Without their continued work and dedication, Node would be an amazing platform that is just too raw to use.

Thanks to Blaine Cook for the idea.

Node.js: From Couch to Mongo

2011-06-30T06:24:31Z

This post is part of a series of articles about my recent experience building Sled using Node.js.

I started with CouchDB and ended with MongoDB.

Working with CouchDB was fantastic. It took no time to learn the REST API and jump right into building the application. I had the first version of the base API ready in 2 days, including full database integration. Couch is a document store using JSON as the document format. Combined with an HTTP REST API, it makes Couch an ideal fit for Node. That’s exactly why I picked it.

I dismissed MySQL from the very beginning for two reasons. First, I had no idea what my schema would look like, and knew I was going to change it a lot over time, as well as allow unstructured data. Second, MySQL is a block database in nature (e.g. database deadlocks, atomic actions, etc.) and while you can use it with Node, it really wasn’t designed for this kind of environment. Also, at the time, there was no quality driver for Node. Oh, and I really wanted to play with a NoSQL database.

I started talking directly to the database but within a few hours switched to use Cloudhead‘s excellent cradle module. The module provides a light layer for managing the HTTP client calls, error handling, simple caching, and common macros for performing multiple database requests in a single function call. It is a very light abstraction layer (and it doesn’t abstract too much either). This worked very well for a few months. I didn’t use cradle’s built-in cache because of the expected size of my database and my constant tweaking of data manually.

Once we started stress testing the server, we hit one of the most common problems with Node’s asynchronous model: the inability to control the execution order of events. We have a simple document which includes a name, place, time, and date. The properties can be modified by multiple people, and if two people change the same property, the last update wins. But when two people change different properties (one the title and the other the place), there should not be any conflict.

To update a document in Couch, you must have its current version. Typically this means going to the database, grabbing the latest version, changing it, and saving it back. The problem is, this simple process breaks into two callbacks, one for fetching the latest document, and another for saving the modified version. When two requests come in at the same time, Node will queue two database fetch requests (for the same document) and then will process each in order. Both requests will return the same document, but after the first changes the document, the second update fails.

I came up with a few ways to address this race condition (I’m sure there are more):

Retry as many times as it is likely to get multiple updates to the same document at the same time. So if we expect 5 people to update the same list at the same time, we should retry at least 5 times.
Use a local cache to store the latest version of recently modified documents and a queue of pending updates. When the first request comes in, the change is added to the queue and the latest document is requested from the database. When the latest document is obtained, the update is applied to both the cached copy and the database. When the second request comes in, the cache can be in one of two states: have the latest document in memory, or pending. If the document is available, it is updated and saved. If not, the second update gets added to the queue. When the document is received, both updates are applied at the same time.
Perform lazy database updates, batching together multiple updates into a single commit. This is performed as soon as there are updates, but it applies all the accumulated updates after it gets the latest copy.
Use another database with native support for partial updates.

#1 works but is butt-ugly. It hard-codes load expectations and makes it harder to identify actual database errors. #2 could work, but I just didn’t feel comfortable with this level of complexity for something as fundamental as database updates. #3 means requests coming in between updates will include data that can be multiple-generations old. So I went with #4 and switched to Mongo.

The good news is that converting the application from Couch to Mongo took two days. I used the powerful native Mongo driver from Christian Amor Kvalheim. Overall the transition wasn’t bad, but it wasn’t trivial. The main difference is how the two databases store documents. Couch stores native JSON objects while Mongo has its own internal format which is similar but not identical to JSON. The biggest difference is Mongo’s support for a more complex set of numbers and native types.

Coming from Couch, I got pretty spoiled at working exclusively with simple JSON documents. JSON-in-JSON-out. If you can make Couch work for you, this alone will make your life much simpler. With Mongo, you have to decide how to handle numbers, dates, identifiers, and other data types. To make the conversion faster, I wrote a small layer on top of the native driver to normalize my JSON objects into and out of the database, converting ids to strings, and all numbers to simple JavaScript numbers.

I also had to add some restrictions on key names not to start with ‘$’ or include periods which are forbidden in Mongo (for a reason). We have an API allowing client developers to store a bit of data on the server for their own use. With Couch, we could let them dump a JSON object as-is (after some security sanity checks), but with Mongo, we had to filter out the forbidden keys and had to move to a key-value store instead.

On the other hand, the database itself became much simpler, removing the need to use Couch views (which meant one less place to manage code). I’m also loving the ability to manipulate arrays within a document with very specific instructions using Mongo’s powerful array update support.

Overall, if you need partial document updates as a basic database functionality, Mongo is a better fit, especially in Node. If you don’t, Couch is simpler and much faster to learn and use. I really love Couch and I wish I could use it. Mongo is pretty awesome too.

One major complaint for Mongo is the lack of decent management tools. Couch comes with the built-in Futon application which is a must-have for any new application development. If I had to use the Mongo command shell to get going in the first few days, it would have taken me twice as long to get going. The available tools for Mongo are awful. They are buggy, they crash, they corrupt data, and when they work, they require you to install Python, Ruby, or PHP on your server – something I refuse to do on my clean Node box.

Node.js: The Style of Non-Blocking

2011-06-30T06:25:41Z

This post is part of a series of articles about my recent experience building Sled using Node.js.

Node is all about non-blocking, asynchronous architecture. This means any activity taking a long time to finish, such as file access, network communication, and database operations, are requested and put aside until the results are ready and returned via a callback function. Instead of asking to read a file and waiting for the operating system to come back with a file handler or buffer, the a callback function is invoked when the operation is completed, freeing the server to handle additional requests.

What gives Node a bit of a negative reputation is how this architecture affects its style of programming and the difficulty some people are having in getting used to it. When I first started, I described this convoluted style of coding like scratching your right armpit with your left hand by twisting your left arm over the shoulder, behind the neck, and under the back of the right armpit. There are days when it still feels like that, but at least now my arms are much more flexible.

Consider the following code, used to fetch a database record and output the user name:

function getUser(id) {
    var user = db.query(id);
    return user;
}

console.log('Name: ' + getUser(432).name);

The function blocks until the database call is completed. This means the server is doing nothing but waiting until the function completes, ignoring other pending requests. In Node, the code is broken into two functions:

function getUser(id, callback) {
    db.query(id, callback);
}

function display(user) {
    console.log(user.name);
}

getUser(432, display);

However, using JavaScript anonymous functions, the code can be streamlined into:

function getUser(id, callback) {
    db.query(id, callback);
}

getUser(432, function (user) {
    console.log(user.name);
});

This nesting of function definitions inside function calls makes the code appear more linear and many find it easier to read. However, it can be tricky for new developers. For example:

getUser(432, function (user) {
    console.log(user.name);
});

console.log('Done');

is going to output ‘Done’ before it outputs the name because the name output waits for the database call to return, place the results on the event queue, and wait for the current executing code (outputting ‘Done’) to finish.

It takes time to get used to this style of coding, especially when your execution isn’t completely linear, but forks based on changing conditions. For example, we have an API call to add an item to a list. The call supports an optional parameter to insert the item at a specific position. Because we store each list item in its own database document, we keep the list order separately. This means, we sometimes make one database call, and sometimes many. At the conclusion, we perform additional processing and return the document id of the new list item.

This produces a very simple blocking code:

function add(list, title, position) {
    // Add new item to 'items' store
    var item = db.insert('items', { list: list, title: title });
    //  Set position if requested
    if (position !== null) {
        var sort = db.get('items.sort', list);
        addToListAt(sort, item.id, position);
        db.update('items.sort', sort);
    }
    // Perform final processing
    var result = { status: 'ok', time: (new Date()).getTime() };
    return result;
}

But not so simple non-blocking code:

function add(list, title, position, callback) {
    // Add new item to 'items' store
    db.insert('items', { list: list, title: title }, function (item) {
        //  Set position if requested
        if (position !== null) {
            db.get('items.sort', list, function (sort) {
                addToListAt(sort, item.id, position);
                db.update('items.sort', sort, function () {
                    finish(item.id);
                });
            });
        }
        else {
            finish(item.id);
        }
    });
    function finish(id) {
        // Perform final processing
        callback({ id: id, status: 'ok', time: (new Date()).getTime() });
    }
}

By embedding the callback function as an argument to the non-blocking function, and separating each logical part of the process into smaller sub-functions, you can code as fast with Node as with any other platform, but it takes some getting used to.

There are a few libraries out there that allow you to code using a blocking style, and turn that into non-blocking code automagically, but I found that to be counterproductive. There is tremendous value in seeing exactly what is happening at run-time and staying close to the execution flow.

Debugging non-blocking code is still very tricky. You can’t just put a breakpoint at the top of the function and step through it. What will happen is that you will break, complete the first request, and then move into the next event in the queue which is not likely to be the next step into your function. Instead, you need to put multiple breakpoints inside each nested function, and rely heavily on logging to analyze your code.

Like any other platform, Node has its own unique style and it can be off-putting to some. But it really doesn’t take long to get used to and once you embrace non-blocking development, the benefits in performance are significant. It also forces you to think hard about your code and architecture, to make sure you are not doing something stupid that will fail-whale you later.

Continue reading from Couch to Mongo.

6 Months with Node.js

2011-06-30T06:15:27Z

For the past 6 months I’ve been spending most of my time building a new service called Sled. I’ll write more about Sled in the coming weeks, but for now all you need to know is that Sled is a collaborative list making tool for small groups.

Sled is built entirely in JavaScript, both on the back-end and front-end. The back-end is build using Node.js (aka Node), the relatively new server-side JavaScript platform. There is some published experience available about building web applications and services using Node, but the experience available is focused on using Node for real-time applications like instant messaging, chat rooms, and games. It is unusual to hear someone recommends Node for building a new web site if it is not focused on real-time.

I expect this to change very fast.

While Sled has a bit of real-time functionality in the form of live list updates from other participants, the core experience is still closer to a traditional web site. We have two main servers, one serving mostly static pages and scripts for rendering by the browser, and an API server. On the client, we use HTML5, CSS, and lot of JavaScript to talk to the API server and display data.

The next few posts are a random collection of things I’ve learned after 6 intense months of working with Node.

To Node or not to Node

To be honest, I jumped at the opportunity to use JavaScript on the back-end. I’m a C++ guy at heart, coming from a long background of building real-time trading systems on Wall Street. I like C++, and I wish I could still be using it, but last time I tried building a web service in C++, it didn’t work out so well… JavaScript has similar esthetics (no meaningful-white-space for me, thank-you-very-much), it is trivial to read by C++ developers, and is very easy to pick up.

Late 2010, I was introduced to Node by Peter Griess who has been an early participant in the Node effort. At the time, Peter was leading an effort to integrate Node into the Yahoo! Mail back-end environment. Once I figured the overall architecture for Sled, I met with Peter to discuss platform options. I knew Peter was working with Node but was reluctant to use it given its early state.

After an hour of talking, it was clear that I should be giving Node a closer look. I envisioned the API server as a completely statelessness machine with a a simple document-based data store requirements, serving REST calls using as much of HTTP as possible. Node sounded like a perfect fit. For the first time in over a year, I got excited about doing web development on a new platform. Don’t underestimate being excited about your technology.

Ironically, when I asked Peter if I could count on him to help out he told me he just gave notice… Great! But he assured me that the Node community is pretty awesome and that getting help should not be a problem. He was right.

My main reasons for choosing Node 6 months ago were:

JavaScript is a natural choice for developers with C/C++ background
JavaScript-all-around makes code management and reusability easier
Team members can easily move between front-end and back-end
Node is the hottest new platform, making it a good fit for a small project looking to attract talent

Given the early stage of the project, I wasn’t too concerned about performance (which proved to be very high) or stability (which hasn’t been a blocking issue, but see below). I was mostly concerned about libraries and shared experience.

My reasons for choosing Node proved to be right. One exception is code reusability between the back-end and front-end which didn’t really proved to be all that useful. We do a little bit of code reusability, mostly for OAuth 2.0 MAC tokens, but even there, the availability of native cryptographic facilities in Node make the reusable code narrower when moving to the browser.

What I didn’t know 6 months ago, but made me stick with Node:

JavaScript is an extremely productive language, especially for back-end development
The Node community is young which leaves plenty of room to impact core components and library development, and the leads are very open to new ideas and suggestions
It’s fun. Super fun.

Is it Ready for Prime Time?

Last year, when I told people I am going to use Node for sled.com, I was greeted with a lot of suspicion, warnings, and skepticism. Six months later I can say with confidence that it was the best decision I’ve made on the engineering side of the project. It got us off to such a great start and that despite the early age of the platform.

As a developer, Node has been amazingly stable. Our server experienced one week of instability in over 6 months due to a race condition bug in the Node HTTP client layer. This was easy to patch and was fixed within a week. I was very surprised at just how stable the API server have been. I’ve started with version 0.2.6 and now running on 0.4.9, but experienced little to no disruptions during the transition from 0.2 to 0.4. The Node core team has done a great job at keeping the project stable, even in this very early stage.

What isn’t so great is library stability.

There are days when I can’t believe some of these libraries were ever executed, and this extends to some of the more popular modules. This is very much a work in progress and it requires a high degree of direct involvement in the library development. If you are going to use a module, you should join its mailing list and keep an eye on what is going on. Being part of the community is absolutely required if you are going to use Node.

The Community to the Rescue

Node has one of the best communities I had the pleasure of working with over the past 20 years. Everyone is accessible, friendly, and eager to help. It doesn’t matter if you are an expert or a novice, you are going to get attention and the help you need. This is especially true for the major module leads. Getting a library patch within hours is a pretty common thing and it helps if you work with them to pinpoint the problem.

All things considered, whatever instability Node introduces, the community counters with plenty to spare.

More Coming

This will be the first in a series of posts about my on-going experience with Node. I hope people new to Node will find it useful. The next post will focus on Node’s non-blocking nature and how it impacts coding style.

Got an interesting Node project going? I’d love to hear about it.

OAuth 2.0 Redirection URI Validation

2011-06-22T06:25:04Z

Why do we require clients to include the redirection URI when exchanging an authorization code for an access token in OAuth 2.0 (section 4.1.3)?

Consider the following scenario:

Evil user starts the OAuth flow on a legitimate client using the authorization code grant type flow.
Client redirects the evil user to the authorization server, including state information about the evil user account on the client.
Evil user takes the authorization endpoint URI and changes the redirection to its evil site.
Evil user tricks victim user to click on the link and authorize access (using phishing or other social engineering methods).
Victim user thinking this is a valid authorization request (it looks kosher), authorizes access. Access is granted to the right legitimate client. So far nothing is wrong.
Authorization server thinks it is sending victim user back to the client, but since the redirection URI was changed, victim user is sent to the evil site.
Evil user takes the authorization code and gives it back to the client by constructing the original correct redirection URI.
Client exchanges the code for access token, attaching it to the evil user’s account.
Evil user can now access victim user data via his client account.

The way this works, the attacker does not get direct access to protected resources, but it tricks the client into attaching the victim’s access token to the attacker’s account.

Pre-registration of the redirection URI can help a lot, but depends on the matching rules. Since many large providers have open redirectors, the attacker can use those to construct a redirection URI that passes the authorization server validation.

Requiring clients to register their full redirection URI without allowing any variations or partial matching is highly recommended.

Closing a Few More Discovery Loose Ends

2010-11-19T21:26:25Z

Two important specifications reach their final stage last month: XRD 1.0 as an OASIS Standard and Web Linking as an RFC. Both are essential building blocks in web discovery and richer distributed web services. It was a long journey getting both of these published in their respective communities – over 2 years of effort for each.

I would like to thank Mark Nottingham for his leadership and hard work authoring the Web Linking specification. This specification will hopefully help bridge the gap between the various formats (XRD, ATOM, HTML, etc.) and provide a consistent way to type links.

Special thanks go to Will Norris for his editorial lead of XRD 1.0, taking my blog posts and notes and turning them into a useful specification with his own added ideas and improvements, and to Drummond Reed for superbly chairing the XRI TC over the past 5+ years and for supporting my crazy idea to drop XRDS and replace it with something simpler.

(All these Brilliant People at) Facebook Make Me Sad

2010-11-03T07:58:25Z

This is not a post about open, about standards, about privacy, or really any criticism of Facebook in any way. In fact, the problem is just how unbelievable the Facebook team is (in a good way). The sheer strength of their talent is almost unmatched in our industry, past and present. The problem is, all that talent is building something I just don’t care about, and no one is left for anything else.

Facebook doesn’t provide me with anything useful.
When it comes to staying connected to the people I care about, they either live with me, I talk to them on the phone weekly, or have an annual dinner when I visit Israel or New York. This is just enough for me. There is a reason why I am not in touch with people from high school, the army, or film school. We all moved on, became different people, changed context, and lost the common thread that united us at the time. My personal Facebook experience of finding long lost friends is mostly a short awkward exchange followed by a one sided stream of useless information.

When it comes to content, I much rather rely on the editorial board of the New York Times for my news, than what my “friends” find interesting. The idea that people I care about are in any way relevant to my news consumption has never produced useful results. When it comes to news, I want to be “friend” with the editors of the New York Times, and when it comes to buying a digital camera, the reviewers at DPReview.

My family and friends are uniformly challenged when it comes to world affairs or digital cameras. Maybe it is just me, but in my offline world, information and sharing works perfectly fine without Facebook. As for casual chatting, games, and leaving comments on people’s photos – I choose to spend my free time doing something else. Instead of dealing with people’s shit, I rather handle chicken shit.

I am in no way suggesting that almost 600 million people are wrong. The massive and highly engaged Facebook user base clearly gets value and satisfaction from the product. I am also in no way critical or judgmental about those who find value there. Good for them! But for me, there are so many other unsolved problems in the world, and they have little to do with social.

Last month at Open Web Foo, sitting outside with about a dozen of some really smart and highly connected folks, I asked people to name three “wow” experiences they had with new web products in the last year. Most had none to share. I asked them to think back 3 years and the list filled up in minutes.

Forget about solving the world’s problem – if you don’t care for Facebook, the web it just boring. It’s stale.

There are many reasons why engineers want to work for Facebook, from the potential windfall to learning just how they are able to ship so much technology so fast. It is an engineering dreamland. But there is one great reason why they shouldn’t: because Facebook will be great without them, but the web might not.

The Growing Web Identity Crisis, Courtesy of Facebook

2010-11-02T21:33:05Z

A concerning trend is showing up in recent TV and print advertisements of companies using their Facebook profile pages as their web identity instead of their own domains. Most of these companies are big corporations with a well-established web presence. Using social networks to connect with consumers and promote brands is not new, but using these identities as the primary corporate web identity is new.

For the past few years, the web identity community has been focused on the engineering side of a distributed web identity platform, developing protocols and tools to improve its usability. These efforts have mostly failed due to lack of clear consumer demand, low value, and an unusable user experience. At the same time, consumers have been eagerly embracing proprietary and closed solutions such as Facebook Connect and Sign-in with Twitter.

While engineers love to debate protocols and solve engineering problems, the core issue has always been the identifier. What should people use to identify themselves to other users and when logging into web services. Such identifier solutions included HTTP URLs (such as your blog), email addresses, proprietary screen names, social networking accounts, PKI certificates, or new naming schemes (there is always a new identity startup just around the corner looking to sell you new moon lots).

The recent trend in corporate web identity is to move away from their own web presence to a page on Facebook or Twitter. This is likely to make deploying a distributed web identity solution even harder. Web users have been voting with their clicks and opted to use their social networking profiles as their web identifiers. This is largely due to a clean, simple, and useful design offered by Facebook and Twitter. But the growing participation of corporations in this space is signaling that consumers are now perceiving these closed platforms as a natural part of the “open” web infrastructure.

What was initially unacceptable – to put another company name on your ad – is now common practice: ‘facebook.com/target’ instead of ‘target.com’. Target no longer fear subordinating their identity to that of Facebook, because for most web users, the ‘facebook.com/’ part is just the new form of ‘http://’ – something they don’t care or understand which comes before the stuff that actually matters. Facebook in this context, is completely transparent to most users.

Facebook has always claimed to be a platform and a utility, and this is the most obvious manifestation of this goal. The real threat to the open web is not from proprietary protocols and closed networks, but from ‘facebook.com/’ and the ‘@’ screen name prefix becoming another well-known prefix like ‘http://’ and ‘www.’.

OAuth Bearer Tokens are a Terrible Idea

2010-09-29T19:09:29Z

My last post about the lack of signature support in OAuth 2.0 stirred some very good discussions and showed wide support for including a signature mechanism in OAuth 2.0. The discussions on the working group focused on the best way to include signature support, and a bit on the different approached to signatures (more on that in another post).

However, the discussion failed to highlight the fundamental problem with supporting bearer tokens at all. In my previous post I suggested that bearer tokens over HTTPS are fine for now. I’d like to take that back and explain why OAuth bearer tokens are a really bad idea.

But first, some live entertainment:

Facebook developer ‘wesbos’ writes:

Hey guys, I’m using the freshly downloaded PHP API

I’ve got eveything setup and when I login to authenticate, I get this error:

Fatal error: Uncaught CurlException: 60: SSL certificate problem, verify that the CA cert is OK. Details: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed thrown in /Users/wesbos/Dropbox/OrangeRhinoMedia/reporting/src/facebook.php on line 512

To which ‘Telemako’ replies (emphasis is mine):

[…] try this one:

$opts[CURLOPT_SSL_VERIFYPEER] = false;
$opts[CURLOPT_SSL_VERIFYHOST] = 2;

And ‘philfreo’ asks:

Is there any good reason why these shouldn’t be a part of the core default $CURL_OPTS?

Well, philfreo, Telemako, and wesbos, you have just turned off one of the most important protections SSL provides: defense against MITM attacks.

Putting All Your Eggs in One Basket

The main problem with using OAuth bearer tokens is the fact that they rely solely on SSL/TLS for its security. Bearer tokens have no internal protections. Just like cash, whoever holds the token is its rightful owner, and proving otherwise is impractical. OAuth 1.0 signatures evolved from the basic requirement not to mandate SSL/TLS. But using signatures goes beyond just a deployment consideration.

In his blog post “defending against your own stupidity”, Ben Adida (a highly respected security expert) writes:

Consider the utility of a safety parachute. A determined attacker trying to kill you will obviously sabotage the safety parachute just as easily as he can sabotage the primary one. So, does that mean you might as well jump without a safety parachute? Of course not. You want to take into account not just the worst-case attacker, you want to take into account your own stupidity. A safety parachute means that, if you packed your primary wrong, you can still live. Defense in depth, as it’s more commonly known in the security community, is usually not about building the 12 layers of security around the “Die Hard” vault that a skilled attacker has to vanquish, one by one. Defense in depth is the humble realization that, of all the security measures you implement, a few will fail because of your own stupidity. It’s good to have a few backups, just in case.

No matter what you do, if for any reason the SSL/TLS layer is broken, the entire security architecture of OAuth bearer tokens falls apart. In a world moving steadily towards multiple means of authentication and verification, this is a step backwards.

SSL/TLS is Harder than You Think

It is common knowledge among security experts that implementing SSL/TLS is notoriously hard. This is why very few attempt to write their own libraries. However, equally well established is that many clients implement SSL/TLS in a way that voids its most valuable defense: that against a man-in-the-middle. If anyone can get between your client and the server, they can capture the token and use it at will.

The example above shows one of two common ways in which SSL/TLS security is compromised. I would love mock the developers quoted but it is a mistake I have made myself three years ago when writing my own OpenID library. I had no problem getting the signatures done, but had a hard time with bad certificate errors. Some were due to local configuration (getting root certificates configured is still a hard thing for most developers to do), but also due to poorly configured servers. My solution, just like the suggestion above, was to ignore these errors.

The problem with SSL/TLS, is that when you fail to verify the certificate on the client side, the connection still works. Any time ignoring an error leads to success, developers are going to do just that. The server has no way of enforcing certificate verification, and even if it could, an attacker will surely not.

This is not new. Ben Adida described this exact issue when reviewing the original WRAP proposal:

But wait, you say, don’t worry, the token is sent over SSL, so it’s all good.

Right. What’s going to happen when someone “forgets to turn on SSL”, which is all too common when security is abstracted out “somewhere down in the stack.” Or when we stop dealing with those pesky certificate errors and just choose not to validate the cert, which will leave the protocol wide open to network attackers who can now literally play man-in-the-middle just by spoofing DNS on a wifi network, capturing the token, and replaying it to access all sorts of additional resources, effectively stealing the user’s credentials.

This might actually be worse than passwords, because at least you can work to educate users about SSL (and after their Facebook account gets hacked, they might actually care), but it’s very hard for users to gauge whether web applications are doing the right thing with respect to SSL certs when the SSL calls are all made by the backend which has trouble surfacing certificate errors.

The second common potential problem are typos. Would you consider it a proper design when omitting one character (the ‘s’ in ‘https’) voids the entire security of the token? Or perhaps sending the request (over a valid and verified SSL/TLS connection) to the wrong destination (say ‘http://gacebook.com’?). Remember, being able to use OAuth bearer tokens from the command line was clearly a use case bearer tokens advocates promoted.

Who is in Charge of Your Service Security?

The main difference between using bearer tokens and signatures is in who they put in charge of enforcing the service security. Signatures ensure that mistakes don’t compromise the token because the secret is never sent with the request. At most, an attacker can use the captured request once, and for enhanced security, signatures can and should be used together with SSL/TLS for a multi-layer protection.

Yes, developers do stupid things, like posting their client credentials (API keys) on public lists and in open source code. But I have yet to see developers expose token secrets like that. It is really hard to expose OAuth 1.0a token secrets by mistake (unless that mistake is leaving your entire database open for public review, in which case, nothing will help you).

Bearer Tokens are not Just Like Cookies

The winning argument in favor of using bearer tokens has always been cookies. Bearer tokens have the same security properties of cookie authentication, as both use plaintext strings without secrets or signatures. However, there is one big difference – the client developer.

For many years, browsers made it insanely easy to ignore bad certificates. It took a lot of time and many attacks to make it hard and scary to approve bad certificate exceptions. Client developers are usually not at the same level as browser security developers. While a JavaScript OAuth client enjoys the same quality code that the browser uses to verify certificates, other clients do not.

The Ironic Solution

We know developers don’t read security considerations sections. That’s a fact of life. Developer only read the parts they need to implement and ignore the parts that ask them to think. Only large companies with dedicated security teams pay much attention to that, and they usually do their own analysis. So warning developers to check the certificate is not likely to accomplish much.

What’s the alternative? SDKs!

If your developers cannot be trusted to implement your API correctly, it is common to offer them a library instead. I hope you can see the irony here. If you solve the SSL/TLS deployment issue with an SDK, why not just implement signatures? One argument used against signatures was that even with libraries, people will try to write their own code. Well, that’s a scary proposition.

Security is Hard

Ben Laurie, one of Google’s top security experts and the OpenSSL project lead writes in his explanation why WRAP “is just so obviously a bad idea, it’s difficult to know where to start”:

The idea that security protocols should be so simple than anyone can implement them is attractive, but as we’ve seen, wrong. But does the difficulty of implementing them mean they can’t be used? Of course not […] every crypto algorithm under the sun is hard to implement, but there’s no shortage of libraries for them, either.

As I have pointed out before, the ‘signatures are hard’ argument dies with Twitter’s recent deployment of OAuth signatures as a requirement.

But Everyone Else is Doing it

My favorite argument in favor of bearer tokens, and the most ridiculous one, is that Facebook, Google, Microsoft, and Yahoo! are all doing it, so that must mean its good. After all, they must have reviewed it and decided it was safe.

The other variation of this argument is that bearer tokens are already widely deployed so we need to support them. All I have to say is quote the obvious: “if your friend jumped off a roof, would you?”

Facebook is using MD5 in their current API, probably because that is what Flickr used when Facebook copied their API authentication scheme. See the pattern? New services simply follow what other established companies are already doing, without questioning the reasoning behind it. Big companies have resources to make these decisions, but they rarely apply as-is to others, and it is simply irresponsible to ignore their leadership position.

This is how we got stuck with Basic authentication and cookie logins. This is what RFC 2617 has to say about Basic HTTP authentication:

The Basic authentication scheme is not a secure method of user authentication, nor does it in any way protect the entity, which is transmitted in cleartext across the physical network used as the carrier.

And yet, it is still one of the most widely deployed authentication scheme on the web. People don’t read specifications, they just follow the market.

Actually, Not Everyone is Doing it

OAuth 1.0 had bearer token support alongside signatures for three years now, and yet, it is barely used. Twitter could have deployed OAuth 1.0 as specified in RFC 5849 section 3.4.4 but chose not to. The claim that bearer tokens are a new feature is false. All OAuth 2.0 does is clean it up and present it in a more accessible way.

The problem is both with the specification and the market hype. By positioning bearer tokens as the default choice for 2.0 developers, and by highlighting the big vendors deploying it, developers are less likely to think for themselves, especially given the allergic reaction many developers have to signatures.

I still have hope that early and future adopters of OAuth 2.0 will make the right decision and not deploy bearer token support for their core APIs. I have never suggested to outright ban bearer tokens. They provide a useful way to experiment with an API, to try out ideas, to test code, and to make API calls that have little to no security requirements. For example, it is perfectly fine to use bearer tokens for any Twitter API call that returns user-specific data that is publically available (like your followers list). Not so fine to use it to update your status or read private messages.

Now What?

I have concluded that the OAuth working group is not a productive place to have this debate. People are much too entrenched in their positions and with existing live deployments, a committee is not the way to reach consensus. Instead, I believe that an open public debate and market pressure is the best way to go. This is where I hope others will join me in asking their service providers to take their security more seriously.

With regard to the specification, I have proposed (and so far received wide support) to make an editorial change that will separate the ‘how to get a token’ part from the ‘how to use a token’ part. This is not a new idea. By doing so, we will enhance the modularity of the protocol and remove any bias for or against bearer tokens and signatures from the specification. Instead, developers will be presented with two or more alternatives, and will have to make their own decision about what is right for their platform. This proposal is still pending working group consensus.

I am sure this will not be the last word on the subject.

More OAuth Nonsense

2010-09-23T06:10:49Z

ComputerWorld is the latest to run a scary story about OAuth 2.0 and how insecure it is. Unfortunately, instead of doing their homework and paying attention to my post, they borrowed a bunch of my quotes (almost half the article), added some original nonsense, sprinkled a few errors, and gave it a sensational headline: “OAuth 2.0 security used by Facebook, others called weak”.

OAuth 2.0 without signatures works just fine for companies like Facebook, because their developers hard-code their API endpoints. There is no danger what-so-ever for an access token to leak or get phished (any more than if they used signatures) because Facebook uses HTTPS for their OAuth 2.0 API. This is why I titled one of the subsections: “Why None of this Matters Today”. My post was about discovery and long term security improvements of the web – not that there is anything broken about today’s implementations.

Let me make this crystal clear: OAuth 2.0 without signatures over HTTPS is perfectly secure for proprietary APIs where the client only sends the access token to a hardcoded endpoint, and uses HTTPS properly.

My favorite part is the quote from Google’s Eric Sachs (who, BTW, is not a co-author of any OAuth specification), explaining why having no signatures is a good thing:

These guys are not experts in security. They can have a hard time understanding security models.

This is exactly the problem. OAuth 2.0 moves some of the enforcement requirements (again, when used with dynamic service discovery) to the client, and the client developer is rarely a security expert. This is why the specification has to assume that the client developer is going to make mistakes and do everything possible to render them harmless.

Raise your hand if you ever wrote a client application using HTTPS and got tired of exceptions about bad certificates, so you just ignored them silently. Unless your HTTPS code is written in a way that completely blocks the connection if there is any error, it might not do you much good. HTTPS is great when done right, and this is just one example of the difficulty in trusting clients to do the right thing.

To be clear, I don’t want mandatory signatures in OAuth 2.0. I think there are valuable use cases for the current bearer token approach. And I agree that developer ease is an important design factor. Signatures were not mandatory in OAuth 1.0a either, but they are an equal part of the protocol, as they should be.

Given the recent support for putting signatures back in (as an optional component), I have pushed back the publication of draft 11 until we can add a signature proposal to the core specification. I expect it sometimes in late October.

Goodbye Open (and Why I’m Staying at Yahoo!)

2010-09-17T22:43:22Z

It’s that time again, to move on. The past three years have been a roller-coaster. Coming from a small startup after a decade in financial services technology, I got to learn, contribute, lead, and provoke open web development. My standards participation landed me a great job, relocated my family to the West coast, and introduced me to a lot of amazing people. It has been awesome.

Over the past couple of months I have been steadily phasing out my open specifications and standards involvement. The OAuth 2.0 core specification is the only thing I am still working on (OAuth is a keeper). Everything else has either fizzled away or lost its interest to me. This should not come as a surprise to anyone who talked to me or read my posts over the past few months.

First, let’s get the most important detail out of the way. I will be working on a new “startup”, building from ground up a new consumer web experience. No, I am not leaving Yahoo! In fact, I’m planning on sticking around for a while. This wasn’t an obvious conclusion, but one that I am extremely excited about (I’ll explain why in a second).

A startup at Yahoo! you ask? Well, you will have to wait a bit longer for more on that. We’re still working out the details. For now, I just want you to know that I am currently hiring a front end developer (in the Bay area). The project is probably going to use Node.JS, CouchDB, HTML5, JavaScript, OAuth, and other cool new stuff. This is a chance to combine startup environment with job security and stability. I plan to write a lot more about working at Yahoo! and this new project soon.

Taking Inventory

In the tradition of recoding my thoughts as I set on a new adventure, these are some of the things I worked on over the past couple of years (and I do mean to brag, a little):

OAuth – Leading author and editor of the 1.0, 1.0a (RFC 5849), and 2.0 specifications, as well as many articles and tutorials. Shepherded the transition from an open community to the IETF. Negotiated licensing terms. Coordinated the handling of a critical security flaw.
Open Web Foundation – First elected president, co-founder, and past chair of the Legal Affairs and Elections committees. Established bylaws, and managed the foundation corporate registration.
XRD – Lead architect and co-editor of the XRD 1.0 OASIS XRI-TC standard (pending). Previously authored the XRDS-Simple specification.
Well-Known URIs – Co-authored RFC 5785 for establishing a URI prefix for well-known locations, and a IANA registry (serving as the Designated Expert approving new requests).
Host-meta & LRDD – Authored a proposed RFC for a method locating host metadata for Web-based protocols, including a unified link-based resource descriptor discovery method. Work coordinated across many communities and standards bodies including W3C TAG.
OpenID – Leading participant in the redesign of the discovery layer and identity services.
Metalink – Served as document shepherded in the IETF for RFC 5854.
WebFinger – Co-author of a simple web-identity layer using account URIs (email-like identifiers) for the discovery of social and personal web identity information.
Portable Contacts – Helped with the initial draft. Facilitated discussions with the IETF vCard community.
OpenSocial – Represented Yahoo! in the foundation creation process. Provided the initial design for the IPR framework and governance model.
Discovery – Participated in specification efforts including Salmon, Activity Streams, OExchange, DiSO, Web Linking, and others, with focus on discovery services and architecture.

Is Open Dead?

Absolutely not. But I think the kind of open I was working on is mostly gone. This is not a bad thing, just the result of the changing industry, people’s careers, and economic conditions. For the most part, the movement that started with OpenID and OAuth is largely over. All the cool kids got grownup jobs and have been mostly missing. Think of the people you used to hear from on a weekly basis, and then try to remember when was the last time they had something new or provocative to say.

I have written extensively about the changing industry and the role of Google and Facebook in the new environment. Standards development goes through cycles of products and explorations. During the exploration phase, people come together with ideas and try to create a new marketplace by working together to invent new capabilities and opportunities. Email, HTTP, and HTML are some well-known examples. During the product phase, companies with specific existing requirements come together to either unseat the market leader, or to solve specific problems where there is no competitive advantage to go at it alone.

Three years ago the social space was a happening scene. Every day a new startup emerged with a new idea on how to make the web more social. Companies focused on specific communities or activities, and there was plenty of excitement. As the space matured, we now have Facebook as the clear winner with a lead that will be impossible to beat anytime soon. Facebook is to social networking what Google is to keyword search. Both are too far ahead of everyone else in their respective areas.

Got Product?

I am sure in a year or two we will go back to the exploration phase, but for now, we are deep in the product phase, and if you don’t have a product, you don’t really belong there. I have not been a fan of Yahoo!’s social strategy for most of the past two years. The vision coming from the top these days is a lot more promising, exciting, and aligned with where I think the company should go, but this was not always the case.

The reality is, that when it comes to the kind of technologies I’ve been involved in, Yahoo! has not been an important player. This did not impact my ability to lead during the exploration phase, but now without a leading product, it is very hard to be effective. If you want to develop new social web standards, you absolutely have to work at Facebook or Twitter. That’s where all the action is. Google might be trying its best, but in practice no one cares about social technology coming out of Google. They have also lost their leadership in most of the groups I have been active in (and not due to lack of trying).

There is still interesting work done by Blaine Cook, Status.net, and the federated social web folks, but it’s all too experimental, and I don’t see it anywhere near mass market anytime soon.

Standards is a Horrible Career Path

Late last year I came to the conclusion that my ability to be effective at my job is coming to an end. Yahoo! has been consistently supportive of my work, provided me with a wealth of resources, and gave me absolute freedom to “make the web better” (that’s how one of my managers described my job). This is very unusual, and something Yahoo! deserves a lot of credit for. Yahoo! repeatedly stepped up to support the community and provided resources without asking what’s in it for them, because it was what was best for the web. This is very rare in our industry.

I was faced with two options: find a new job where I can be effective again, or move on. Given my passion (and success) at developing open specifications and communities, I spent a considerable time and effort looking for a similar opportunity elsewhere. I talked to all the usual suspects, got some offers, got ignored a lot, and at the end arrived at the conclusion that it’s time to move on from standards. I do want to say that of all the companies I talked to, I was impressed with how respectfully Facebook handled the interaction. Can’t say that about some of the others.

I also realized just how destructive choosing standards as a career path can be. The standards world is very demanding and will suck every free minute you have. Most people contribute very little, and at the end, a handful of people end up carrying all the load. The problem is, no one wants to foot the bill for those suckers. Only a handful of very large corporations (mostly telecommunication and hardware) support employees doing standards full time, and mostly to serve their self-interest.

Anyone who ever met me knows that I am not the right person to represent a company line. I have my own voice, it’s loud (sometimes obnoxious), and I’m not shy of using it. I don’t know of any other company that allows its employees to speak their mind so freely and openly, even when they are in disagreement with the company’s official policy. Even our legal team has policies in place to enable that. I have often been out of alignment over Yahoo!’s OAuth and OpenID strategies. But at no time was I told to shut up, something at least one of my collaborators at Google cannot say.

At the end, especially during hard economic times, finding a role focused on standards is hard. It is especially hard when you have views that are not likely to always be in line with those of your employer. Kids – stay away from standards as a career.

It’s All About the Product

I miss building stuff. I miss writing code and seeing how something starts from nothing and progress into a working system. With the exception of the last three years, I have been building applications since I was 8 years old, using my first Sinclair ZX81 computer. When I was 11 a friend and I wrote a palm reading program. We got into trouble when we set up a stand at a local fair and gave someone a printout stating that her intelligent was below average. Hey, not my fault her head-line was so short.

More than OAuth, my passion the past few years has been discovery. It is very rewarding to see some of my work finally making its way to actual products. But most of the time it was just frustrating, putting all this effort in without any actual commitment from those building products. I still think discovery is going to be more and more critical as the web grows and becomes more distributed, but it will take a while longer before my vision can be fully appreciated.

Why Yahoo!?

That’s a great question, and one that I have been debating for almost a year now. It was a hard decision, but once I reached it, it was clearly the right decision, and I am super excited about the coming year. To get to my conclusion, there are a few factors you need understand. When considering a new employer, I mostly care about: their products, their culture, my manager, my team, and how they balance work-life. Technology, compensation, location, reputation, and IPO prospects have never made a difference to me.

So let’s go through them one by one.

Products –

Yahoo! is still working to define what it is. I have heard the official description but like most people, am still trying to figure out what it really means. What I do know is that we have the biggest audience on line, a long list of best in class properties, a leading brand name with unparalleled trust (no one is afraid of Yahoo! trying to take over the world), and a user profile that is exactly the people I want to build products for – normal everyday web users. If you get the core Yahoo! user base to use a new experience, your impact is massive – far beyond just web habits.

Facebook, Google, and Twitter are all one hit wonders. Granted, these are some unbelievable wonders, and I have nothing but awe and respect for what these three companies are doing in their space. But let’s face it, they are known for one thing (Google has a few new promising areas like Chrome and Android), and working there most likely means focusing on improving an existing product rather than building something new.

I rarely use Twitter or Facebook. They don’t add much value to me. That’s in no way criticism of their products, just that I care a lot more about what people say on BackyardChickens.com than on Twitter or Facebook. Same goes for the people in my family. One of the reasons Facebook is such a success is the fact their employees are truly passionate about the product and use it intensely, and I think that that alignment is important.

Working for Yahoo!, especially these days, offers more opportunities to work on something new and exciting than anywhere else. It offers unmatched access to resources and audiences, and a management team that is eager to hear new ideas from every level of the organization.

Culture –

If it wasn’t clear by now, I value openness, directness, flat hierarchy, equality, diversity, and individuality. This list alone rules out a bunch of companies. Google’s approach is to build an army of (really smart, talented, and somewhat socially inadequate) clones. When you are hired, you don’t know what you are going to work on. Google’s hiring process is designed to make sure they hire great engineers that can do anything. This allows them to shift resources around as needed quickly, but in the three times I engaged Google over a job, I could not get past that. I will not take a job where I am just another engineer.

Facebook is much better, but their emphasis is mostly on coding skills alone. I could not figure out what Twitter’s culture is like (and I’m not sure if they know either).

Yahoo! scores high on every one of my cultural values. The problem at Yahoo! is that the culture has been taken hostage by complaining, ranting, sarcasm, irony, and a generally depressing mood. The underlying culture is very much there, but it is hard to enjoy it given the general atmosphere. Part of the problem, is the constant hammering of the tech press, making a big deal out of every departure (much more than at other companies).

This is where Yahoo!’s collegial culture is counterproductive to its own best interest. Consider some of the recent high profile departures. Yes these are all smart, talented individuals. But with a few exceptions, these are mostly people who got pass on in a recent reorganization, failed to deliver a product, did not fit in a company focused on mass market consumer experiences, were offended that someone else got a better role or more control, or been there for more than 5 years. In other words, this is corporate business as usual.

The problem is, because Yahoo!s are so nice (people often wonder why they keep me), they will never leak out and share these negative realities to balance the news (we are known to leak plenty of other stuff). This is great for the individuals working there, because at some point, we all move on, but at the moment, it is taking a toll.

I am in no way blaming the press for any of the underlying issues, or for doing their job highlighting them. I don’t think the coverage is fully balanced (“People are running away from Yahoo!” vs. “Facebook stole another Google engineer”), and it is clearly one of the main factors casing the current mood. I only mentioned this as an example of how the culture and mood are linked.

My solution is simple. Two months ago I decided to try and stop complaining. No more sitting at URL’s (our café) bitching about everything we suck at. For now, it seems to work.

My Manager –

I think most people will consider this entire post as one big ass kissing. I hope those who know me see this for what it is – a public venting of my views, frustration, admiration, and random rants. I like to share my views, even when others find them inappropriate. It has made me plenty of enemies, but no one has ever claimed not to know where I stand or where they stand with me.

Keep in mind that I’m the guy who, on his first week at Yahoo!, was featured on Valley Wag under the headline: “New Yahoo: Joining up without severance package would be like ‘running into a burning building’” (I am still the number one search result for ‘running into a burning building’). With that said, I am going to simply state that my manager, Mike Curtis (head of Mail engineering), is awesome. I recently had the rare opportunity to pick my own manager and I think that says it all. I am also happy about the chain of command between me and the CEO.

My Team –

My discussions around a new role outside Yahoo! did not reach the team evaluation stage (with one exception at Facebook, where the team was great). So while important, it was not a big factor. What was, is my ability to build my own team, which is something I have always enjoyed doing. At Citibank, I twice built large teams building high frequency trading systems, and it is something I miss doing. I am not going to be building anything this big anytime soon, but I get to hire a couple of people today and that’s plenty fun.

Work / Life Balance –

This is Yahoo!’s number one asset and number one liability. If you are an employee, you will not find a better place to have a family, to have a hobby, or in other words, to have life beyond work. Between company events, benefits, and overall atmosphere, Yahoo! is the most welcoming place I have ever worked (or interviewed with) when it comes to having a family (our twins are turning 6 next month).

Beyond its culture, the reason for that is the profile of the employees. Being one of the few “old” web companies around, and currently attracting less young blood than its competitors, Yahoo!’s employees tend to have a family and are not the kind of people usually joining a startup. The problem, of course, is that this makes it harder to move fast and innovate at a competitive rate. Every time I visit Facebook I feel like (at 36) I’m the oldest guy in the room – the place is like a college campus.

But with the new management’s determination to innovate, and to give anything a try to accomplish that, I really believe Yahoo! is in a unique position to attract a large group of brilliant engineers, those not interested in the bootstrapping startup life, but innovative and passionate about building the next generation of web experiences. Being the place where you don’t have to choose between spending time with your family and changing the world through web innovation, is something Yahoo! has a unique opportunity to become, more than anyone else. We are clearly not there yet, but this is something I want to work on.

This is also why I believe Yahoo! has a better chance of understanding our audience, the same way Facebook truly understand theirs. Our company profile is very much the audience we seek to serve – everyone.

On to the Next Great Thing

It took me a while to get here, and after a yearlong search, I’ve decided to stick around and it feels right. There are still more things to figure out about my new project and the company, and like any “startup”, anything can happen and this can be short lived or a huge success (or anything in between). Many people I talked to these last few days told me I’m nuts, day dreaming, or naïve. Maybe I am.

But I’m having more fun than everyone I know, I got a great job, I’m paid well, my employer truly cares about my well-being, I get to spend quality time with my family, and I even get to play farmer in my spare time (chickens, ducks, bees, large garden and an orchard).

Can you say that, without qualifications, about your job?

I’m hiring a front end developer (you can start tomorrow), and Yahoo! has plenty of great jobs available in both new and core products. If you are interested, and available in the Bay area, email me at eran@hueniverse.com.

This blog will continue to highlight open standards, because that’s something I am still very much passionate about, but it will also offer more insight into life at Yahoo!, my new project, and whatever else is on my mind. This is going to be yet another fun ride. Stick around.

OAuth 2.0 (without Signatures) is Bad for the Web

2010-09-16T06:07:00Z

OAuth 2.0 drops signatures and cryptography in favor of bearer tokens, similar to how cookies work. As the OAuth 2.0 editor, I’m associated with the OAuth 2.0 protocol more than most, and the assumption is that I agree with the decisions and directions the protocol is taking. While being an editor gives you a certain degree of temporary control over the specification, at the end decisions are made by the group as a whole (as they should).

And as a whole, the OAuth community has made a big mistake about the future direction of the protocol. A mistake that is going to make OAuth 2.0 a much less significant agent of change on the web.

All this Crypto Business

OAuth 1.0 allows application developers to sign requests. Signatures remove the need to send plain-text secrets on insecure (or secure) channels. Instead of sending the secret with the request for the other side to compare against their copy of the secret (similar to how passwords work), the secret is used to calculate a value which cannot be converted back the secret itself. Instead, the signature can be verified by someone with a copy of the secret.

When performing an irreversible calculation that the other side can verify, signatures protect secrets by simply never sending them on the wire. By removing the need to send secrets, applications don’t need to rely on other protocols (such as SSL/TLS) to protect the plain-text secrets. That’s not all signatures provide, but more on that later.

To sign a request, developers have to follow a list of steps in a very specific order and with much care (which often feels like battling a dragon). The smallest mistake causes the entire request to fail. While the OAuth 1.0 signature process could have been somewhat simpler (no double encoding, different sorting, no URI parsing into query parameters, etc.), any time developers need to canonicalize data, stuff breaks. Even beyond the complex math, cryptography is hard because it is generally unforgiving. It does not tolerate mistakes.

WRAP and the Stupidity Threshold

After deploying OAuth 1.0, many companies discovered the cost of supporting OAuth 1.0 due to mismatching signatures. OAuth 1.0 looks simple enough for developer to code from scratch instead of using a library (as opposed to SSL or TLS which no one in their right mind will try to write from scratch). When developers write their own code, they are likely to get one small detail wrong. It didn’t help that the specification was vague and implicit about many important details (which since has been corrected in the RFC).

Ironically, the bigger the company was, the more resources it had, and the least interesting and useful API it offered, the louder the complaining about OAuth signatures was. This had an easy and straightforward solution: provide better libraries to your developers as well as better (or any) debugging tools. Alternatively, make your API so valuable that developers will be motivated to struggle through it and figure it out. Unfortunately, this was not the solution the people behind WRAP had in mind.

At the heart of the WRAP architecture is the requirement to remove any cryptography from the client side. The WRAP authors observed how developers struggled with OAuth 1.0 signatures and their conclusion was that the solution is to drop signatures completely. Instead, they decided to rely on a proven and widely available technology: HTTPS (or more accurately, SSL/TLS). Why bother with signatures if instead the developer can add a single character to their request (turning it from http:// to https://) and protect the secret from an eavesdropper.

Much of the criticism that followed focused on the fact that WRAP does not actually require HTTPS. It simply makes it an option. This use of tokens without a secret or other verification mechanism is called a bearer token. Whoever holds the token gains access. If you are an attacker, you just need to get hold of this simple string and you are good to go. No signatures, calculations, reverse engineering, or other such efforts required.

As Secure As a Cookie

WRAP was based on a simple, and powerful argument: bearer tokens are already a core web architecture. While far from ideal, the WRAP security model was directly based on cookies – the authentication layer behind almost every web application. Why bother to create something more secure if it makes it harder for developers to use, while not actually improving the overall security of the service. As long as a site offers both an OAuth API and a human web interface (i.e. a web site), the overall service will only be as secure as its weakest part – the cookie-based authentication system.

The problem with this argument is not today, but 5 years from now. When trying to propose a new cookie protocol, developers will make the same argument, only this time pointing the finger at OAuth 2.0 as the weakest link. Removing signatures and relying solely on a secure channel solves the immediate problem, and maintain the same existing level of security. But it lacks any kind of forward looking responsibility, and the drive to make the web more secure. It’s a copout.

What makes this more frustrating is that the people behind WARP are some of the brightest security minds on the web. These guys know exactly what they are doing, and it’s not like they don’t care. They just gave up and decided that the best they can do is maintain the status quo. They are also representing a large and powerful coalition of big companies too lazy to work a little harder by helping their developers use signatures successfully.

Doesn’t HTTPS Solve Everything?

HTTPS guarantees an end-to-end secure connection. The implementation and deployment details are critical to ensure that, but when done correctly (which is not always the case), is a great solution. What HTTPS provides is a secure channel. Any secret, password, or bearer token sent over HTTPS is protected and cannot be compromised by an attacker listening in on the line. HTTPS allows a client to send a secret to its desired destination securely.

However, HTTPS can’t help if the client’s desired destination is a bad place. HTTPS doesn’t help prevent phishing attacks because anyone can get an SSL certificate and show the secure icon in the browser. The fact you are using a secure channel doesn’t mean the entity on the other side is good. It just means that no one else can listen in on it (just the bad guys). If a client sends their bearer token to the wrong place, even over HTTPS, it’s game over.

Another issue is that the OAuth working group could not even reach consensus on actually requiring HTTPS, leaving it as an recommendation for services to decide. Even OAuth 1.0 requires HTTPS for its plain-text flavor, which was added to get it published as an RFC. Instead, OAuth 2.0 is satisfied with just a warning. OAuth 2.0’s solution is to allow (but not require) access tokens to be short lived. By limiting the bearer token’s lifetime, stolen tokens are only useful for a short period of time, limiting the potential damage.

Why None of this Matters Today

OAuth today is used together with proprietary web service APIs. There is little to no interoperability across these services (Facebook API used only on Facebook, etc.) and almost no clients performing discovery of any kind. Because the API endpoints are hard coded into the client, when combined with HTTPS, there is no risk of leaking the tokens. In this setup, the client does not need to do much thinking about where to send tokens and how to protect them.

Unlike cookies which are sent to the server based on a somewhat complex set of client rules, OAuth clients today don’t use any rules. Instead they use a single token for an entire service with all API endpoint preconfigured. There are no new subdomains to handle, or really any kind of unexpected or dynamic interaction. In this environment, bearer tokens over HTTPS are just fine.

Why All of this will Matter Soon

As soon as we try to introduce discovery or interoperable APIs across services, OAuth 2.0 fails. Because it lacks cryptographic protection of the tokens (there are no token secrets), the client has to figure out where it is safe to send tokens. OAuth reliance on the cookie model requires the same solution – making the client apply the security policy and figure out which servers to share its tokens with. The resource servers, of course, can ask for tokens issued by any authorization server.

For example, a protected resource can claim that it requires an OAuth access token issued by Google when in fact, it has nothing to do with Google (ever though it might be a Google subdomain). The client will have to figure out if the server is authorized to see its Google access token. Cookies have rules regarding which cookie is shared with which server. But because these rules are enforced by the client, there is a long history of security failures due to incorrect sharing of cookies. The same applies to OAuth 2.0.

Any solution based on client side enforcement of a security policy is broken and will fail. OAuth 1.0 solves this by supporting signatures. If a client sends a request to the wrong server, nothing bad happens because the evil server has no way of using that misguided request to do anything else. If a client sends an OAuth 2.0 request to the wrong server (found via discovery), that server can now access the user’s resources freely as long as the token is valid.

It is clear that once discovery is used, clients will be manipulated to send their tokens to the wrong place, just like people are phished. Any solution based solely on a policy enforced by the client is doomed.

No Discovery for You

Without signatures, OAuth 2.0 cannot safely support discovery. It is a waste of time and a risky business. Clearly, the OAuth community today does not care enough about discovery and interoperable services to do something about it. The cryptographic solutions proposed so far are focused on self-encoded tokens and other distributed systems, based on narrow use cases promoted by the likes of Google, Microsoft, and a few other enterprise-focused companies.

Without discovery, smaller companies will have a harder time getting their services accessible (e.g. when importing your address book from any provider, not just the big four).

Now What?

I am not advocating throwing OAuth 2.0 out, starting over, or requiring signatures. All I have ever advocated for is the inclusion of a basic signature option in the core specification, in the spirit of OAuth 1.0. The 1.0 signature isn’t perfect, but as the Twitter developer community demonstrated, is clearly within reach.

Twitter a Hot Princess, Google an Empty Castle

2010-09-16T06:06:05Z

Over the past two years I have been arguing that the problem with supporting OAuth 1.0 signatures wasn’t with the signatures, but with the lack of value in trying to figure them out. The primary argument made by the WRAP authors and now the majority of OAuth 2.0 contributors is that signatures are hard and developers are stupid. This combination, they argued, is costing them developers.

To address this, they argued that the only solution is to remove signatures. I countered that instead of creating a new protocol, the companies complaining (primarily, Google, Microsoft, and Yahoo!) should invest in quality libraries and debugging tools.

My point was (and still is) that if you give developers value, they will fight to figure out the signatures. A couple of weeks ago Twitter discontinued their support for Basic authentication, and what these people said cannot happen, happened. All these developers figured out how to migrate their application to OAuth 1.0. That despite the lacking Twitter developers support, alleged bugs, and other complaints about Twitter’s implementation.

Conclusion

Don’t expect knights to battle dragons if your castle is empty. Twitter put a hot blond (or brunette) princess (or prince) in their castle, and their (API) knights fought the evil (signature) dragon and got their reward. Google and the rest of the big web providers with their useless offering of boring APIs left their castle empty.

Guess what! The kind of knights who come to fight dragons living in empty castle are there for the fight, not to do something useful. Yes, battling dragons is a bitch, but knights tend to forget that once they get their happy-ever-after. Give them an empty castle and they will do nothing but obsess about their battle scars.

Why does this matter? Because without signatures, there can be no secure discovery and less open web.

How to Enter a Standards Working Group

2010-09-06T06:16:57Z

As the OAuth specification editor, I am approached daily with comments, questions, ideas, and some very odd solicitations. It is quite fun. It seems that most people joining the work are determined to undermine their contribution by doing their best to alienate or insult the community, and often the editor.

This is not unique to OAuth – I have seen the same mistakes in many other places. Since working on an early draft of the OAuth 2.0 specification, I started collecting tips for newcomers. Here are some that would make your entrance into a standards working group much smoother and productive.

Don’t start with a proposal, start with a question.

Assume you don’t understand everything, especially if you have feedback or ideas for changes. Instead of jumping in with some new re-architecture of the entire protocol, start by asking why it is the way it is. Engage the community first with a question, and work your way to change requests based on the answer you receive. By asking a question, you allow for the possibility that whatever if bugging you has a valid reason.

Keep your message as short and to the point as possible.

You are an unknown so people will look for reasons to ignore your message. Most people don’t even open messages from newcomers. Instead, they wait to see if a thread develops. In other words, they wait until someone else puts in the extra work to deal with the new guy. If your message is short, it is more likely not to be ignored.

Don’t email the editor privately, asking for permission before asking the list.

You don’t need permission to ask questions or post to the list. All the specification editors I know (and certainly me) are usually busy and work in bursts. This means your email is most likely going to be moved to some queue of questions and feedback about the specification, to be processed right before the next editorial cycle. You are also wasting their time by forcing them to read your message twice (once directly and once on the list), and in many cases, repeat the discussion. This is why we have a public list – just join and post away.

Don’t prefix your message with I’m new here.

It’s never a good idea to start a relationship with an excuse. The only reason to put an “I’m new here” disclaimer on your message is if you were too lazy to do your homework, read group archives, read the full specification (and revision history), and think before writing. If you do all that, your contribution is as valid and relevant as that of anyone else. The working group regulars know you are new.

Focus on what you know best.

Pick an area where you have experience and provide feedback to show you know what you are talking about. Don’t start with a multi-page review of the entire specification. Instead, focus on the areas you are most knowledgeable about and demonstrate your skills and experience through the quality of your contribution. Don’t provide a summary of your career – saying you have many years of experience is the least effective way to get that message across. Let you work speak for itself.

Never start with an editorial suggestion.

Editors like to write, and they take pride in their work. They also have their own unique voice. Starting a relationship with a specification editor by criticizing their style is not a good way to get them to listen. Typos, mistakes, and other technical issues with the specification are always welcomed, but offering replacement text or ideas on how to say something differently will most likely be ignored, no matter how good the suggestions are. Wait for an editor to ask you to suggest text before sending in your own revision.

Start with a compliment, show good progress.

Pick something you think is good about the work, and highlight it. This isn’t rocket science, just common sense. Don’t compliment any one person in particular (like the editor or chair). Compliment the work and the community and do it in a way that adds value, like highlighting what is strong and good about the work, providing additional support.

Clearly state your role and introduce yourself.

Spend a few words to introduce yourself the first time you write something. Don’t just barge in – it’s rude. Also, standards are highly political, and in a political environment, people want to know who they are dealing with. No need for life story, just the basic facts (lurker, developer, representative for a big company). And don’t assume everyone know who you are just because you think you are important.

Do your political homework.

Identify the key participants and align yourself with those you agree with, showing you are well versed in the internal politics. Joining the discussion through agreement with a primary participant allows you to establish a work relationship with those most likely to be in a position to listen and impact. Agreeing with someone is usually better than disagreeing – it adds emphasis to an existing view, and doesn’t cost you political points going directly against someone else.

Make sure to be right if you are going to keep a thread going.

If you are going to stand your ground and keep an argument going, against the group consensus, make sure you know what you are talking about. It is perfectly fine to keep an issue going if you are unhappy with the resolution, but if you are new, this behavior is more likely to alienate the group than win your argument. If you are going to keep at it, make sure to add new facts and new examples to make your point.

Don’t explain basic standards principals.

Assume everyone knows what bike-shedding, design by committee, or KISS are. Don’t lecture the working group about process and other philosophical ideals. For most of the people involved, this is probably not the first standard.

Don’t prefix more messages with “I’m still new here”.

You didn’t need to state it the first time, so please don’t keep repeating you are (still) new. Yes, we got it, you are new here.

All This Twitter OAuth Security Nonsense

2010-09-02T22:53:58Z

In a wordy article that could have been much shorter and a lot less sensational, Ryan Paul of ArsTechnica throws mud mostly at Twitter, but saves plenty to throw at OAuth. Unfortunately, Ryan Paul (who clearly is a smart guy) is heavy on the accusation but light on the arguments. Typically, I would go over this article an item at a time, but I’m right in the middle of draft 11 of OAuth 2.0 which is a much better use of my time. If you want to read a great rebuttal, Ben Adida’s response (as always) is a great read.

The OAuth standard has many significant weaknesses and limitations. A number of major Web companies are collaborating through the IETF to devise an update that will fix some of the problems, but it’s still largely a work in progress. The current version of the standard—OAuth 1.0a—is an inelegant hack that lacks maturity and fails to provide clear guidance on many critical issues that are essential to building a robust authentication system.

First, the current version of the standard is RFC 5849. The RFC not only changed a lot of the terminology, highlighted the known security limitations, and has completely new prose, it also explicitly clarified at least one of the author’s issues regarding the handling of timestamps (which most companies don’t even bother to check anyway).

My favorite silliness from the article, when discussing the lack of specification details about using client secrets in installed applications:

Part of the problem is that the specification doesn’t provide much guidance about what implementers should do instead, which has forced them to improvise. Facebook and Google Buzz have both come up with reasonable solutions and offer desktop-appropriate OAuth authentication flows that do not require a secret key or require the end user to go through a complicated copy/paste process.

The specification is very clear (as the article quotes) – don’t use client secrets in installed applications! The reason why the specification doesn’t say much more is because there is no solution. It does not exist for a distributed application unless you issue a different secret to each installation. To say that Facebook and Google came up with reasonable solutions is pure sensationalism. What is their reasonable solution? They don’t use client secrets. In other words, they do exactly what the specification says.

The article does bring up important points implementers should pay attention to when using OAuth, such as the secrecy of their client credentials, the exact details of their user experience, how they authenticate the user (cookies, etc.), and an overall awareness to phishing. But OAuth, just like other security protocols, is designed to be implemented by security experts. In addition, there are simply no widely available solutions for many of these problems.

If Twitter uses the client secret in installed application for anything other than gathering statistics, well, they should reconsider. But it’s not like they have other alternative. That’s the only valid “news” the article has to offers.

It’s easy to throw mud without getting dirty with making a fully baked technical argument – and it makes for a fun read too. But when it comes to a widely deployed security protocol, scoring page views by scaring readers about security is not fair game. It is always valuable to highlight OAuth’s weakness, but in context, with the right security risk analysis, and with clear comparison to other alternatives.

I expect more from ArsTechnica.