Jason Crawford

“Request for Comments”

2013-11-30T22:53:47-08:00

Great story about the origin of “Request for Comments” as the title for Internet standards documents:

In the summer of 1968, a small group of graduate students from the first four host sites—UCLA, SRI, UC Santa Barbara, and the University of Utah— had met in Santa Barbara. They knew that the network was being planned, but they’d been given few details beyond that.…

A month or so after the new group began meeting, it became clear to [Steve] Crocker and others that they had better start accumulating notes on the discussions. If the meetings themselves were less than conclusive, perhaps the act of writing something down would help order their thoughts. Crocker volunteered to write the first minutes. He was an extremely considerate young man, sensitive to others. “I remember having great fear that we would offend whoever the official protocol designers were.” Of course, there were no official protocol designers, but Crocker didn’t know that. He was living with friends at the time and worked all night on the first note, writing in the bathroom so as not to wake anyone in the house. He wasn’t worried about what he wanted to say so much as he wanted to strike just the right tone. “The basic ground rules were that anyone could say anything and that nothing was official.”

To avoid sounding too declarative, he labeled the note “Request for Comments” and sent it out on April 7, 1969. Titled “Host Software,” the note was distributed to the other sites the way all the first Requests for Comments (RFCs) were distributed: in an envelope with the lick of a stamp. RFC Number 1 described in technical terms the basic “handshake” between two computers—how the most elemental connections would be handled. “Request for Comments,” it turned out, was a perfect choice of titles. It sounded at once solicitous and serious. And it stuck.

“When you read RFC 1, you walked away from it with a sense of, ‘Oh, this is a club that I can play in too,’” recalled Brian Reid, later a graduate student at Carnegie-Mellon. “It has rules, but it welcomes other members as long as the members are aware of those rules.” The language of the RFC was warm and welcoming. The idea was to promote cooperation, not ego. The fact that Crocker kept his ego out of the first RFC set the style and inspired others to follow suit in the hundreds of friendly and cooperative RFCs that followed. “It is impossible to underestimate the importance of that,” Reid asserted. “I did not feel excluded by a little core of protocol kings. I felt included by a friendly group of people who recognized that the purpose of networking was to bring everybody in.” For years afterward (and to this day) RFCs have been the principal means of open expression in the computer networking community, the accepted way of recommending, reviewing, and adopting new technical standards.

From Where Wizards Stay Up Late: The Origins Of The Internet

Amazon's “two-pizza teams”: The ultimate divisional organization

2013-07-30T08:00:00-07:00

Amazon’s “two-pizza teams” are well-known; they’ve been written about in Fast Company and the WSJ. But almost everyone misses the point. They aren’t about team size—they’re about autonomy and accountability.

For context, here’s a succinct explanation from Ben Thompson of the difference between divisional and functional organizations:

In a divisional organization, different products are companies unto themselves. They have their own marketing, their own engineering, and their own finance. There may be some centralized functions, such as legal and HR, but everything that makes money for a product lives within that product’s organization.

Most crucially, each product has its own profit-and-loss statement (P&L). The performance of each division is thus clear to everyone from the CEO to the division presidents to Wall Street, and accountability is usually directly tied to the P&L.…

Functional organizations are the exact opposite: each function is siloed, and products cut across functions in a matrix-like fashion. This significantly expands the role of the CEO and his leadership team in all products, as that is the main point of coordination.

In this model, there is usually no ownership for a product’s P&L. Instead, everything accrues to the company’s all-up P&L, including compensation.

Ben points out that most multi-product companies, including Microsoft until recently, organize divisionally. But Amazon took divisional organization to the extreme with two-pizza teams (2PTs, for short).

Two-pizza teams are so named because they’re small: 6 to 10 people; you can feed them with two pizzas. The most important aspect of a 2PT isn’t its size, though, but its fitness function. A fitness function is a single key business metric that the senior executive team (the “S-team”) agrees on with the team lead. It’s the equivalent of the P&L for a division: a single metric to provide focus and accountability. In some cases, the fitness function is literally a P&L: for instance, when I was on the SEM team, we used the contribution profit from the sales driven through sponsored links, minus the cost of those clicks. In other cases, it’s something more clever: e.g., teams that built fulfillment center software would have metrics related to the efficiency of picking, packing, and sorting.

Once approved, the team is then free to execute relatively autonomously to maximize its fitness function—to pursue creative strategies and to set its own internal priorities. A handful of engineers, a technical product manager or two, and maybe a designer all report directly to the 2PT lead (or “2PTL”); there’s no need to coordinate even across teams, let alone across divisions, to get something done. This model has helped Amazon stay nimble and innovative even as it has grown.

It has also helped the company attract and retain entrepreneurial talent. The 2PTL job is similar in character to being general manager of a division—just as a GM is like a CEO of a product within a company, a 2PTL is like the CEO of narrower function—but the job is small enough in scope that a junior manager can take it on. This makes it an attractive opportunity and a strong growth experience for an ambitious young leader with ownership and drive—core Amazon cultural values.

Essentially, Amazon found a way to take the idea of divisional organization, and push it all the way down the management hierarchy to the level of small teams with first-level managers.

The divisional-vs.-functional question doesn’t necessarily have a single right answer for all companies. The post quoted above mentions Apple as a (rare) example of a company that thrives with a functional org. But I think that the most successful companies find an answer that fits their mission and culture, and then take it to the extreme. Amazon did.

Epilogue: This post describes Amazon as I knew it when I worked there 2004–2007. People who were there more recently tell me that since then, fitness functions themselves have been abandoned, although the ideas of accountability, autonomy and ownership are still core and very strong.

Thanks to Andrew Miner, John Rauser, Eric Vadon, Ben Bernard, and Blake Scholl for commenting on a draft of this post.

Discussion on Hacker News

Follow me on Twitter

Meteor Demystified

2013-06-12T07:47:25-07:00

In which I open the hood of the Meteor framework and see what’s inside.

I recently started a new web project, and several friends suggested I check out Meteor, a new framework for building real-time webapps. Their intro screencast is impressive, and the project generated a lot of excitement when it was first announced. It lets you write real-time apps focusing just on your domain model and views, without all the plumbing—like Rails did for regular websites several years ago.

At the same time, the prospect of using Meteor for a production app would make any battle-scarred devops veteran nervous. The current version as of this writing is 0.6.3, which they call an “early preview”, saying: “Meteor is still under rapid development. Expect major API changes in each release.” (When will it hit 1.0? “More than a month, less than a year.”) The list of projects still has a lot of toy/demo apps; it’s unclear if anyone uses it in production on a major site.

Moreover, Meteor is “magic.” How does it work? What is it really doing? Will you even be able to understand it?

Despite all this, the prospect of rapid development of a real-time app was so seductive that I decided to look into it thoroughly—in effect, to take it apart and see how it works—to get underneath the magic.

What follows is a summary of what I learned from this exercise, which I haven’t seen explained quite this way anywhere else.

Meteor’s central goal #

To understand Meteor you need to understand its central goal, which drives the design.

The core problem Meteor solves is a data synchronization problem. Your webapp is a distributed system, with data on both client and server. The main challenge is to synchronize the data instantly and seamlessly between the two—and more, to synchronize all clients with each other. Ideally it should appear to all users that they are interacting directly with a single, shared store of data.

To solve this, Meteor gives you:

A local copy of the data model, easily accessible via lookup and query interfaces. (This is common to all webapps that take the approach of client-side MVC.)
An open socket to the server to get real-time updates (over WebSockets, long polling, or a similar transport. This is common to all real-time apps.)
A latency-hiding mechanism that makes user updates appear to happen instantly. The method is clever: The core business logic exists on the client as well as on the server, and when the client sends a request, it also runs that logic locally, without waiting for the response—in effect simulating the server and predicting the outcome. (This is possible because JavaScript is the language on the server as well as the client, so the same code can run in both places.)

This is an optimistic protocol: in the normal case, it hides all the latency of the network request; if there is an error or race condition, the client will get the authoritative response from the server soon enough. However, it doesn’t preclude security: the server is still the authoritative source of data; the client has no extra privileges. And you can share only a subset of the server code with the client, if you want to keep some of it private.
A dependency-tracking mechanism to automatically refresh views when their underlying models change. In most client apps, the view or controller observes its model in order to refresh the view when the model is updated, but usually these dependencies must be wired up by hand. Depending on the app, this can be tedious and error-prone. Meteor provides a subsystem to track these dependencies automatically.
A sophisticated template management subsystem that allows you to refresh or re-render views at any time without disturbing the user. For instance, the state of all inputs is preserved, so if the user has entered text it is not lost on a refresh. This makes it safe to do refreshes frequently and automatically by the mechanism just described.

The first three of these together mean that all interactions are local; the sync to the server, in both directions, is decoupled from user interaction. The last two mean that the view is kept in sync with the model by default, with minimal developer effort.

All together you get the central goal: a distributed app with the look and feel of a local one.

What you get: The parts list #

With that context, let’s open the box and see what’s inside.

Here are the key pieces Meteor comes with. All of this is described in the documentation, which I’ll link to where appropriate:

A web server on top of Node that lets you deploy some pieces of code to the server, deliver other pieces to the client, and have some that are in common to both (described here).
An RPC framework, over WebSockets or long polling, that does latency hiding as described above.
A MongoDB API that is unified across client and server. The client-side implementation uses the RPC framework to forward the calls to the server, and thus takes advantage of its latency-hiding mechanism.
A pub/sub mechanism that implements the Distributed Data Protocol, an open standard defined by the Meteor group.
A template management subsystem as described above.
A dependency-tracking system as described above that is used to autowire views to their data dependencies. This is nicely decoupled into an independent component, so you can use it for anything, not just views and models.

Read the docs to learn more; they are clear and helpful.

Concerns #

This technical deep dive had a paradoxical effect on me.

On the one hand, I became much more comfortable with Meteor’s “magic” once I understood how it worked. On the other hand, once I understood it, the gap between Meteor and existing web technologies appeared to shrink. After all, plenty of realtime apps are built on Node and MongoDB using REST APIs and a pub/sub package like Socket.IO. At that point, the only parts of Meteor you’re really missing are latency compensation, dependency tracking, and template management. Each is a solvable problem (especially if you don’t have to solve the general case).

In that light, some of my concerns about Meteor came to the fore:

It’s not battle-hardened. Again, it’s an “early preview”. It hasn’t had time to bake in production—to test performance and correctness in the field, to resolve subtle incompatibilities with other popular packages and libraries, to determine what configuration options need to be exposed.
Scaling. As of this writing, Meteor hasn’t really solved the horizontal scaling problem. As far as I understand, there’s no server-side pub/sub component yet (such as Redis), so once you get beyond one app server, you’re polling the database for updates. Fixing this is on the roadmap for 1.0 but not currently being worked on.
Latency. When I deployed one of Meteor’s example apps to their own servers, there was noticeable latency on first load and even while navigating around a small, simple dataset (see for yourself). (This may be an artifact of Meteor’s hosting environment, which is only meant for demos and experiments, but I couldn’t deploy to Heroku to test that hypothesis: Heroku doesn’t yet support WebSockets, and Meteor doesn’t currently have a way to turn them off.) Improving app load time is also on the roadmap, but not until after 1.0.

And that brings us to some of what is odd about the design of the Meteor server. In its enthusiasm for a new way of writing apps, Meteor has left behind some of the basic architecture of the web. It fully embraces the single-page app concept, delivering the entire client bundle on each request (which contributes to the latency problem). There is not yet a way to break up an app into pages: the server actually delivers the same response to the client no matter what URL was requested; like a native iOS app, it’s up the client to interpret the URL and display the appropriate view. Meteor does not provide a REST API; it doesn’t even really use HTTP (except to deliver the initial client bundle), sending all requests and responses over a two-way connection in RPC fashion.

Indeed, a Meteor app is not truly a web app; it is an RPC-based client/server app that happens to be deployed over HTTP and run in a browser.

I think this belies a certain philosophical leaning on the part of Meteor’s designers—but it doesn’t need to become a holy war. The Meteor group is not opposed to a REST API or a page model; in fact, these things are on the roadmap for 1.0. They’re just not there yet.

Overall, Meteor is an impressive piece of engineering, but I’m not going to use it for my current project. Instead, I’m going with old, boring technologies that have been around for years and are already used in industrial-strength applications. Not that Meteor won’t be ready for prime time eventually—the team is smart and accomplished, and I wouldn’t bet against them. But in matters of production software, I’m just old enough and just crusty enough to stick with the devil I know.

Thanks to David Crawford and Andrew Miner for commenting on a draft of this post.

Discuss on Hacker News.

Immigration is a moral issue

2013-04-24T09:04:05-07:00

I’m glad to see many of my colleagues in the tech community promoting immigration reform (e.g., FWD.us). They point out that immigrants are often innovators and job-creators, and that immigration helps our economy. This is true and important, and it’s a good reason to loosen our draconian immigration laws.

But there is more that can be said—that must be said—in favor of open immigration. Immigration is an issue of human lives, of individual dreams and ambitions, of personal happiness and love.

Every immigrant is a human being with unalienable individual rights. They want to come here to pursue a better life. Their personal goal might be to teach at one of the world’s best universities, to work at one of the world’s best companies, or to pursue art in a country that gives freedom to artists. Or perhaps they just love our spirit of individualism and want to build their lives here no matter what they do.

We take those goals and we bury them under a mountain of paperwork. We take dreams and we put them on years-long waiting lists. We take futures and subject them to arbitrary caps and quotas. We take love and place it under the review and approval of government bureaucrats.

By what right? By what standard? There is no other form of bigotry in this country still practiced, still institutionalized against so many people. If we had special work permits just for women, it would be denounced as sexism. If we had caps or quotas on the number of blacks living in the country, it would be denounced as racism. But in the name of a misguided protectionism, we impose both of these on immigrants for no crime other than having been born abroad. We make exceptions for skill, for money, for intelligence—as if merely existing inside our borders were a privilege to be bought or earned.

And what of the US citizens who want the immigrants here? Have we no rights? For every worker we keep out of the country, we hurt the company and the team he would have joined. For every teacher we block, we hurt the students whose lives he would have touched. For every artist we deny, we hurt all the fans who would have loved his work. And what of the immigrant and the American who fall in love?

We should look at our immigration laws not merely as an economic inefficiency, but as a moral outrage. We should look at them with indignation and disgust.

In the name of individual rights, we should dismantle the entire bureaucracy of immigration restrictions. Open the doors. Let them in. Short of a threat to public safety, no immigrant should be denied entry to the US or residence here for any reason. Nothing less is morally conscionable.

Ask not what immigrants can do for our country. Ask: by what right, by what privilege, can anyone presume to keep them out?

Thanks to Ben Bayer and Manjari Narayan for commenting on a draft of this post.

Discuss this post on Hacker News.

Fighting regulation and winning

2013-04-06T14:31:00-07:00

Regulation is often the enemy of the entrepreneur. It can be used by incumbents as a weapon to protect themselves from new competitors, or it can prevent consumers from taking prudent risks on new products and services. Which is why I’m fascinated by stories of entrepreneurs fighting regulation and winning.

I’ve been collecting these stories. Here are a few just as instruction or inspiration for any entrepreneur faced with a regulatory battle:

Most recent was a WSJ profile of Travis Kalanick’s and Uber’s struggles against local transportation agencies. When Uber moves into a city, the local government and incumbent competitors often claim that Uber is violating rules—or try to get the rules changed to shut them out, in some cases explicitly on anti-competitive grounds. Uber responds by rallying public opinion and, most importantly, by not backing down:

Last year came the clash with regulators in the city where they order red tape by the truckload: Washington, D.C. A month after Uber launched there, the D.C. taxi commissioner asserted in a public forum that Uber was violating the law.

This time Uber was ready with what it called Operation Rolling Thunder. The company put out a news release, alerted Uber customers by email and created a Twitter hashtag #UberDCLove. The result: Supporters sent 50,000 emails and 37,000 tweets.… The taxi commission complained that the company was charging based on time and distance, Mr. Kalanick says. “It’s like saying a hotel can’t charge by the night. But there is a law on the books, black and white, that a sedan, a six-passenger-or-under, for-hire vehicle can charge based on time and distance.”

In July, the city tried to change the law—with what were actually called Uber Amendments—to set a floor on the company’s rates at five times those charged by taxis. “The rationale, in the frickin’ amendment, you can look it up, said ‘We need to keep the town-car business from competing with the taxi industry,’ ” Mr. Kalanick says. “It’s anticompetitive behavior. If a CEO did that kind of stuff—you’d be in jail.”

What Uber is facing—incumbents trying to protect their business by wielding the club of regulation—is the same thing Nextel faced over 20 years ago. Chris Dixon tells the story:

In the late 80s and early 90s, the FCC’s rules banned more than two cellular operators per city. As Nextel’s cofounder said, “the FCC thought a wireless duopoly was the perfect market structure”. Nextel (called Fleet Call at the time) circumvented these rules by acquiring local (e.g. taxi, pizza truck) dispatch radio companies, which they then connected to create a nationwide (non-dispatch) cell phone service.… Predictably, the cellular incumbents tried to regulate Nextel out of existence.

In this case, the FCC had the power to kill Nextel, but decided not to. Nextel took a chance, and won.

Sometimes, though, striding forward and hoping the powers that be rule in your favor is too great a risk to take. In that case, there’s only one thing to do: Get the law changed first. That’s what Naval Ravikant did with the JOBS Act:

With an army of lobbyists and industry leaders at his side, Ravikant set out to convince lawmakers that the JOBS Act, which eases securities laws, would encourage more small business investment. Any one who’s seen a debate or political speech over the past four years knows that “building small businesses” is golden rhetoric for politicians on both sides of the aisle. But even amid this climate, it wasn’t easy.

“People told us that it was impossible – and it actually basically is impossible,” Ravikant said. “We just pulled out all the stops.”

By pulling out the stops, that means encouraging entrepreneurs and investors to call their Congressmen to communicate the importance of the legislation. “We basically showed (lawmakers) how in every state and in almost every major city, we had a bunch of investors and a bunch of startups that were willing to stand up for what we were doing.”

Any startup facing an existential threat from regulation should remember these stories. The laws of the land are not laws of physics: they can be changed. Deliver a clearly better product or service, rally public opinion, argue your case—and follow this advice from Kalanick:

“Stand by your principles and be comfortable with confrontation. So few people are, so when the people with the red tape come, it becomes a negotiation.”

Thanks to David Crawford, Keith Schacht, and Blake Scholl for commenting on a draft of this post.

Discuss this post on Hacker News.

Plan for some failures

2013-02-25T00:12:52-08:00

An interview Hunter Walk did with Stewart Bonn about his time at Electronic Arts contains some tactics for experimenting and still delivering at high quality.

First, undercommit:

When I was GM of EA Studios, my job was to deliver an agreed upon number of games on a particular schedule and at high quality. In order to make sure we delivered those products, we had to have more titles in development than we would commit which would give the Producers the freedom to slip some products if they needed more time as long as they could deliver other titles sooner. I was able to change the expectations of the company such that only 70% of the titles in development were committed.

Second, overbudget:

I also encouraged the producers to “kill early and kill often”. Entertainment software is a “hope business” as in “I hope all this time and money produces a hit product.” Sadly that is not always the case. In fact, the titles most in trouble take the most of everyone’s time in an attempt to improve them. Killing a bad project frees up more than its share in time and energy. To encourage people to kill stinkers, I gave each producer a budget for write-offs so they could kill a title and not have the company feel any unplanned financial impact.

In other words, plan for some failures—both in your schedule and your budget. Set expectations that not all experiments will succeed, and you’ll be free to focus on the winners and make them great.

Discuss this post on Hacker News.

Cowboys & artists

2013-02-21T00:13:00-08:00

As a software development manager I’ve met two types of engineers. I call them cowboys and artists.

Artists love the beauty of the systems they create. They dislike maintaining legacy code, preferring to start from a clean slate—a blank canvas. Given this opportunity, they create works of elegance.

Cowboys revel in their ability to wrangle any system, no matter how obfuscated. They are happy to wade into a mess and bring order to chaos. They are the rangers, the buckaroos, the heroes who save the day.

These types are not completely at odds. The best artists are perfectly capable of cleaning up any mess, but they don’t enjoy it—the effort leaves them drained, whereas a true cowboy is exhilarated by the adventure. Similarly, the best cowboys produce excellent, clean code when starting from scratch. I have met excellent engineers of both types, and gotten along with each very well. I’ve also seen them get along with each other very well.

But as an engineering manager, it’s important to understand these types and know where your team members fall. Artists are happiest if you allow them to work to the highest standards, preferably starting from nothing. Cowboys feel most rewarded if you recognize their skills at untangling the worst messes and credit them as the Navy SEALs of the team.

A good manager will be able to work with either. A good team could probably use both.

Thanks to Andrew Miner for commenting on a draft of this post.

Discuss this post on Hacker News.

Working with Steve Jobs

2013-02-10T11:31:00-08:00

An article by Glenn Reid on “What it’s Really Like Working with Steve Jobs” is worth reading because it counters a false image of how visionary leaders work.

First, even the best product people don’t design by divine revelation. Reid says has this to say about building product:

It is a process which requires understanding the parameters, the goals, and the gives and takes. Stretch what’s possible, use technologies that are good, rein it in when the time comes, polish it and ship it.… It wasn’t magic, it was hard work, thoughtful design, and constant iteration.

Second, great leaders—even those with highly developed intuition and excellent judgment—don’t dominate by sheer force of personality, with their own ideas winning out simply because of who they are. Reid says:

There was kind of an approach we took, unconsciously, which I characterize in my mind as a “cauldron”. There might be 3 or 4 or even 10 of us in the room, looking at, say, an iteration of iPhoto. Ideas would come forth, suggestions, observations, whatever. We would “throw them into the cauldron”, and stir it, and soon nobody remembered exactly whose ideas were which. This let us make a great soup, a great potion, without worrying about who had what idea. This was critically important, in retrospect, to decouple the CEO from the ideas. If an idea was good, we’d all eventually agree on it, and if it was bad, it just kind of sank to the bottom of the pot. We didn’t really remember whose ideas were which—it just didn’t matter.

Jobs was a master product designer and a forceful personality. But the way he and his team worked was the way any team should work—iterate constantly and let the best ideas win.

Thanks to Andrew Miner, Blake Scholl, David Crawford, and Keith Schacht for commenting on a draft of this post.

Discuss this post on Hacker News.

How to not micromanage

2013-01-31T08:51:00-08:00

One day in February 1966 [Bob] Taylor knocked at the office of ARPA’s director, the Austrian-born physicist Charles Herzfeld, armed with little more than this vague notion of a digital web connecting bands of time-sharers around the country. At any other agency he would have been expected to produce reams of documentation rationalizing the program and projecting its costs out to the next millennium; not ARPA. “I had no formal proposals for the ARPANET,” he recounted later. “I just decided that we were going to build a network that would connect these interactive communities into a larger community in such a way that a user of one community could connect to a distant community as though that user were on his own local system.”

After listening politely for a short time, Herzfeld interrupted Taylor’s rambling presentation. He had followed his young associate’s theoretical research closely enough to know already the gist of his ideas. All he had was a question.

“How much money do you need to get it of the ground?”

“I’d say about a million dollars or so, just to start getting organized.”

“You’ve got it,” Herzfeld said.

“That,” Taylor remembered years later of the meeting at which the Internet was born, “was literally a twenty-minute conversation.”

(From Dealers of Lightning: Xerox PARC and the Dawn of the Computer Age)

Discuss this post on Hacker News.

Why you shouldn't hire based on experience

2013-01-29T00:18:00-08:00

Because experience doesn’t necessarily make people better at what they do:

Extensive research in a wide range of fields shows that many people not only fail to become outstandingly good at what they do, no matter how many years they spend doing it, they frequently don’t even get any better than they were when they started. Auditors with years of experience were no better at detecting corporate fraud—a fairly important skill for an auditor—than were freshly trained rookies. When it comes to judging personality disorders, which is one of the things we count on clinical psychologists to do, length of clinical experience told nothing about skill—“the correlations,” concluded some of the leading researchers, “are roughly zero.” Surgeons were no better at predicting hospital stays after surgery than residents were. In field after field, when it came to centrally important skills—stockbrokers recommending stocks, parole officers predicting recidivism, college admissions officials judging applicants—people with lots of experience were no better at their jobs than those with very little experience.

This doesn’t mean you can’t learn from experience. It means that learning from experience is not automatic. Some people can make the same mistakes over and over again and be none the wiser for it.

This is why I don’t classify engineers by experience: mobile vs. web, iPhone vs. Android, Ruby vs. Python. I look for intelligence, drive, fundamental skills, and an ability to get stuff done. Six months into the job, a great engineer who started with no experience on a platform will outperform a mediocre engineer who started with several years’ experience.

Discuss this post on Hacker News.