subbu.org

Code the Infra

2013-04-14T01:25:10Z

Code the infra. There is no other way to make operations predictable and repeatable. The opposite of coding the infra is what I call as “box hugging”. If you log into boxes to configure, install packages, start/stop services, or do any maintenance, you are a box hugger. Coding the infra requires that you treat automation artifacts (shell scripts, puppet manifests, fabric scripts etc) and configuration as code. If you’ve no repeatable code to bring up bare infra into a desirable operational state, then you are a box hugger. Box hugging is a bad habit, and is bad for the business. It makes recovery from failures time-consuming. It does not scale with needs. Most fat-finger and admin cockup related outages start with box hugging. Sure, it may have worked 100 times, but just one fat-finger mistake is enough to make your team’s life miserable.

Two steps to cure box hugging – first, internalize the idea that the box you’ve just finished setting up meticulously is going to burst into flames the very next minute, second treat operations the same way as you would treat software development.

Coding the infra is not hard.

1. Treat infra as ephemeral

Infra is not permanent. It will fail. You can estimate MTBF with some assumptions, but failures won’t follow estimates. MTTR is more important that MTBF. When you treat infra as ephemeral, the act of bringing up new infra to a desired operational state becomes a normal and known practice. You ignore the dead nodes, and focus on bringing up new nodes as quickly as possible.

2. Think of system setup as a series of state changes

Start with the basic infra, and apply a sequence of steps to change the state of the infra to bring it to a desired state. The steps could be installing packages, configuring them, starting servers, setting cron jobs and so on. This is no different from most coding exercises – start from a known state, apply some computations, and arrive at a new state.

3. Make the steps repeatable

This is like coding any math problem. You first solve it on paper to arrive at an algorithm. Then you would code the algorithm so that you can repeat it every time you need to solve the same math problem again. It is the same with operational changes. It might seem time-consuming to treat operations this way, but unless you make the steps repetable through automation, you can’t recover from failures easily. Repeatability is a way of rehearsing recovery. Node died? Cool – just run the automation to bring up a new node. You’re back in business.

4. Implement idempotency

Repeatability alone is not sufficient when the state changes are numerous. You need to make each state change idemotent. Apply the same change again – the system should not burn up. Practicing idempotency makes the outcome certain. If something breaks in the middle you can replay the whole sequence of changes when you know that each step is idempotent.

5. Review, test and version control

Finally, apply the same engineering rigor to automation artifacts as you would apply to software development – that is, ensure that the automation scripts are peer-reviewed, tested and maintained in source control. There should be no difference.

DevOps is not just about integrating dev and ops, but is about treating operations as development, and development as operations.

Getting Paid to do What you Love

2012-12-31T15:59:16Z

Just imagine being able to get group of so-called "individual contributors" together and build software with freedom to execute in a manner that works best for the team, with minimal or no project management overhead, no architect(s) or committees to ratify design decisions, and the empowerment to quickly change the course as necessary. There is just one catch – the team must be after a mission to accomplish some measurable and timely business goals. How the team gets there is up to the team. I've had opportunities to form teams and execute in this manner twice so far in my 20 months at eBay, and I love the experience. What's better than getting paid to solve problems for real customers and scale without the overhead and slowness that is typically attributed to large companies!

During this course we blended a number of ideas from open source projects so that we get the best of the both worlds.

Having been in this trade for years, I've come to realize that how a team organizes itself and conducts its activities is a critical contributor to the quality and agility as well as the enthusiasm and happiness of the team members. One of the ways to get a team of people behind a set of goals with passion and enthusiasm is to let the team decide the how. Unfortunately, that does not happen in most places of work.

Let's imagine that there is a mission and set of goals to accomplish. The typical model I've seen used goes like this – One or more architects (or some senior equivalents) define how the big picture ought to look like to meet those goals. An architecture is drawn as pretty diagrams and published into slide decks and wikis. The big picture is presented and discussed, and then is broken into parts that smaller teams can execute. Smaller teams are then formed. Finally coordinators like project/program managers get assigned to conduct the development process so that dependencies and communication can be streamlined. This model creates an hierarchical organization supposedly to help reach the goals.

Then Conway's Law happens. First, the organization will start to influence and dictate choices. Second, since each team is going after a part of the mission, the parts may not necessarily guarantee an integrated experience for the customer irrespective how well they were built. Third, coordination of the development of parts creates an extra layer of friction and slowness that most productive engineers hate. I've seen things that take 10 minutes to fix get discussed between teams for hours and days only to get dropped in the end.

Such an organization is counter-productive.

How are successful open source projects organized? The rules of engagement in open source model are not organization-driven but are organic:

Individuals come from different parts of the organization, often working in different timezones. There is no direct mapping from managers to things that engineers work on.
Individuals get to work on the things they work on because they have established their credentials with the rest of the group, and not because they work for such and such part of the organization or they've been assigned to.
Decisions don't get taken because some senior person said so. In stead, individuals have the power to make, and the responsibility to socialize and defend their choices.
There is no one paid to coordinate. Open source projects don't have the luxury of support and coordination roles. Certain members may volunteer to coordinate things like prioritization, and releases but it usually takes a small amount of their time. Such roles are often rotated.
Inferior ideas are voted out as the group tends to recognize innovators and strong executors – community recognition is a key perk in open source projects. Consequently, you don't get to hide behind a manager curtain and build stuff in isolation.
There are no mundane tasks. Consequently, things that are manual or that require inferior skills don't exist.

All these are easy to apply even in non-open source projects irrespective of the size of a company, and it is well worth it. But it opens up an important question since you've some business goals to accomplish and work for a man.

How do you make sure that things that are needed to be done get done. In a community-centric model, some work items may attract no attention.

This is easily solvable.

To start with, the team must get behind the mission and stated goals and make the mission their own. What I've learned is that, when individuals are given the ownership and freedom to execute and a mission, they usually get things done end to end. You don't need to tell them how. Things fall through the cracks when they don't have that freedom. Of course, this is contingent on creating a reward structure that rewards meeting the end goals and not specific tasks or actions.

In a system where anyone can join and pick up anything, mundane things won't get done because those are tedious and repetitive. This usually leads to a subpar team structure with lower tiers of engineers doing mundane tasks. The only way to avoid this is replace mundane tasks with clever solutions. Create those solutions such that you can make yourself redundant from doing those ever again.

Finally, let the mission and goals help decide what needs to be done and what does not need to be. Feature creep happens when teams are looking at parts and not the big picture. You can minimize this by making every one part of the big picture.

The net result is a team that is focused on the mission with the freedom to figure out how to organize and conduct itself. This may sound anarchic, but I would argue that is it is objective.

But does it work when you have hundreds of developers all working on a large project, or any project for that matter? I don't know but I think it is important for the people involved to take the time to figure out.

Milestone.js – Project Tracking for GitHub Projects

2012-12-28T22:06:42Z

We use GitHub Enterprise at work for almost all of our activities at work. In addition to maintaing source in git, we also use GitHub for code reviews, issue tracking, milestones, incident reports etc. However, what we lack most are (a) an ability to quickly track project progress, and (b) someway to track large features that can take one or more milestones to complete – like Launchpad‘s Blueprints. Here is how Launchpad describes a blueprint.

Blueprint is a lightweight way to manage releases of your software and to track the progress of features and ideas, from initial concept to implementation.

Enter Milestone.js – the idea of Milestone.js is to fill these gaps. It grew out of our needs at work.

First, it uses GitHub APIs and Timeline JS to map milestones on a timeline so that you can find what is happening in any given milestone.

Second, it follows some simple conventions to manage blueprints:

Create each blueprint as an issue, and label it as a “blueprint” – it does not matter what you label it as.
Create issues for all work items for each blueprint.
For each blueprint, add the work items as comments starting with {issue_id}:.

Milestone.js will then show all related work items for each blueprint. Here is an example.

HTTP Pipelining is a Hit and Miss

2012-07-21T00:09:41Z

The idea behind pipelining in HTTP 1.1 is to allow clients submit multiple idempotent HTTP requests over a connection to a host, let the server to process them in parallel and respond in the order in which the requests were made.

In theory, pipelining has some benefits:

Increased parallelism on the server – this seems obvious since the client is able to dispatch many requests over a connection.
Reduced amount of the time an HTTP request is waiting to be made in the browser – in stead of queuing up requests behind one another, the browser can dispatch them as soon as it knows that a resource needs to be fetched
Improved connection reuse – as a consequence of increased parallelism and reduction in queued times, it is arguable that pipelining would result in a reduction in number of connections that the browser needs to open.

These may result in reduced latencies. However, finding data with consistent patterns to prove benefits of pipelining is hard to come by. The first set of charts below show total time requests were in the queue for each connection. The red circles show requests made without pipelining and green circles show requests made with pipelining enabled. The size of each circle is proportional to the number of resources requested for each connection. The results show 100 runs made from a pool of private instances of WebPagetest on Windows XP with Firefox over DSL profile.

Any pattern showing that pipelining reduces queued times is absent in these charts.

The second set of charts below show the number of connections used by the browser for each run. The red lines show runs without pipelining, and green lines show runs made with pipelining.

Again there is no consistent pattern in these charts to show the pipelining improves connection reuse.

The point is here is that though the idea of pipelining is simple in theory, making tangible benefits out of pipelining may be hard.

Your MVC may be Slowing Down Your Site

2012-07-03T18:56:13Z

One of most common design patterns used on the Web is the model-view-controller. Though some variations of this pattern exist, the basic idea is simple – a controller parses and processes the incoming HTTP request, thinks for a while, and then writes a view as HTTP response.

The point of this post is not to blame MVC, but to point out that implementations in web frameworks may be leading to higher time to first byte times.

But did you know that, this style, as commonly implemented, may be slowing down your site? See the waterfall charts below for some of the Alexa's top 500 sites in US, captured using webpagetest over the DSL network profile with Firefox.

When you look from the network perspective, all these charts share a common pattern – deafening silence, followed by a log jam.

The network is silent when the server is thinking (the green area). As soon as response starts to trickle down (the blue area), the browser starts new network activity and immediately gets into a log jam mode. In this mode, the browser is busy making DNS look ups, TCP handshakes, followed by several HTTP requests over a limited number of connections.

From a user experience point of view, this is bad as it leads to a rapid backlog of HTTP requests. As the number of connections is limited, it takes a while for this log jam to clear up which increases the overall page latency. A protocol like SPDY can deal with the log jam better, but reducing the jam is better for the user.

The problem here is not with MVC per se, but how it is commonly implemented. Most often, the thinking part consists of talking to back-end systems and/or some computation completely blocking response generation. Even in implementations that do some of this thinking asynchronously, view rendering ofter starts after the thinking part is done. This contributes to higher "time to first byte" times and the subsequent log jam.

Below is a waterfall chart of a site that minimizes this log jam.

In this case, the server is starting HTML delivery as soon possible so that the browser can start to download some of those resources sooner. The HTML delivery continues even after some of the subsequent resources complete. This is possible by interleaving HTML delivery with backend work and stream chunks of response as they become ready.

Though such sooner delivery does not completely avoid the log jam, it does help ease it.

If you do MVC, don’t forget to think from the network point of view.

Update: See the Hacker News thread for more comments: http://news.ycombinator.com/item?id=4194500

QCon NY Slides on “Making Things Work Together”

2012-06-20T16:28:09Z

Here are the slides from my QCon NY talk where I presented ql.io as a way to interop across servers.

Making Things Work Together

View more PowerPoint from Subbu Allamaraju

See qcon2012.pptx for the original deck.

Want SPDY?

2012-06-27T16:04:28Z

Web performance is a complex topic with many contributors to poor latency. Of all the bottlenecks to site speed, HTTP traffic interests me the most. We know that both Google and Twitter run their primary domains on SPDY for performance reasons. But does this approach make sense for other sites? Here is my observation from the data collected for a very small set of domains – SPDY makes most sense for domains that meet two criteria – those that serve a large proportion of resources on a page, and those that consequently make the browser put requests in the “blocked” state for the longest time. For other domains it may not matter, which most likely explains Guy Podjarny‘s observations in Not as SPDY as You Thought earlier this week. SPDY’s multiplexing can help minimize blocked times.

See the charts below for some popular sites. Each chart shows two bars for the top five domains for the resources in a page (including the page’s domain itself) that meet the above criteria. The blue bar shows the number of resources per domain, and the red bar shows the average number of milliseconds that a request to each resource was in the “blocked” state. For a given resource, the difference between the time a connection is made to the time a request is made is the “blocked” time. You can find blocked times in HAR files. You can use a simple visualization tool I wrote to draw these charts for your HAR files.

Though a large blocked time does not necessarily mean that the user perceived latency is poor, it can be a contributor to latency particularly when the network is bad – like the time I captured this data. I captured the data for these sites when I was on a Hotel wifi network – I got lucky a couple of days ago as the hotel I stayed at had very low download speeds, and I could not get any other work done.

Here are my observations from these charts:

In most cases, domains serving large number of static resources contribute to most of the blocked time. This is prominent on sites that change the content frequently or for each request (such as search, news etc.) due to their low cacheability.
In most cases, such domains are hosted on homegrown or commercial CDNs, and so enabling SPDY on those domains will be a key contributor to SPDY adoption.
Serving root domains of such sites over SPDY may not help, but serving those resource intensive domains over SPDY might.
Undoing some or all of the domain sharding via SPDY capable domains will be necessary to take advantage of SPDY. This is because SPDY requires one connection per domain, and domain sharding may force the browser into making multiple connections to the same host.

Note that my tests are not scientific and my sample size is tiny. Caveat emptor!

Code Coverage Games

2012-05-07T20:33:38Z

You can easily game this. But not this.

Protocols vs Styles

2012-05-03T03:41:19Z

My RESTful Web Services Cookbook has been out for nearly 26 months. From Feb 2010 till the end of 2011, O’Reilly sold 13,052 copies (both print and eBook format combined, per the royalty statements received to date) of this book. Given that O’Reilly’s eBooks are DRM free, I would like to think that several thousands more copies were downloaded for free on torrent sites, but there is no way to know.

Writing this book was a painful exercise. I spent most of 2009 writing the drafts – at that time Yahoo allowed me to spend about 20% of my time for nearly 4 months to write the first draft (for which they own the copyright of the book). It was painful because REST was about style. Not having seen all aspects of the style work in reality, I was not fully convinced about parts of the style. Separating style from protocols was a time consuming yet learning exercise. How could I say that following such and such Roy Fielding’s constraint leads to evolvability without dissecting evolvability into concrete details? The last thing I wanted was to write a book that paints a nice picture with the benefits one would get out of REST without discussing the specifics. I wanted to avoid handwaving at all cost.

In these two years, I changed quite a bit.

I no longer actively engage myself in discussions about REST.
I unsubscribed out of rest-discuss in 2011.
I don’t even subscribe to Roy’s idea that REST APIs must be hypertext-driven or that “A REST API should spend almost all of its descriptive effort in defining the media types”.

What I learned in these two years were the following:

Don’t confuse styles with protocols.
Understand what you are going to interoperate with and make tradeoffs as necessary.
Interoperating between style deviations is more important than arguing for style adherence.
Interoperability is partly a protocol problem and partly a software problem.

Architecture styles are guides, but protocols specify rules. Roy’s REST codifies (to some extent reverse-engineeres) the Web into a set of constraints. Constaints are what they are – guides sans context. They are subject to tradeoffs. They postulate benefits when applied. But protocols are different. They set ground rules for different systems to work together. Just like you don’t start driving on the left side of the road (in the US), you don’t break TCP or HTTP’s messaging formats. When you break such protocols, you prevent networked systems from working together.

But we tend to mistake the style (i.e, REST) for the protocols. I made that mistake several times (shame on me!) in the past, and I still see it being made. I took care to avoid this while writing the book, but I may not have succeeded fully. Strong belief in a style often leads to zealotry. This is no different from the zealotry we see in socio-political debates.

Yet not all Protocols are the same. I tend to view protocols making up a pyramid. Where a protocol stays in this pyramid shows how widely it helps interoperability.

    /\
   /  \
  /    \
 /______\

Towards the bottom are protocols that strive for the widest range of interoperability possible. TCP is an example. HTTP is another example. As you go up the pyramid, the amount of relevance and influence a protocol carries becomes narower. For instance, I would keep RFC 2616 towards the bottom of the pyramid, but an RFC like 5023 (AtomPub) towards the top. AtomPub’s reign of influence was limited then and is even now. I would similarly place a media type like application/json towards the bottom, and a type like application/my-grand-fathers-facebook-profile+json at the top. A perspective of where a protocol belongs in this pyramid certainly helps ration energy.

So, would I write this book differently now? I certainly would.

I would change the title to something along the lines of “Recipes to Build Client Server Apps Using Web Protocols”, though the editor might ask me to change to include some current trending words.
I would emphasize even more on protocols.
I would cut down or clearly demarcate the parts of the book that suggested design styles (such as designing formats).
I would spend a big chunk on how to interopere without necessarily agreeing on the style.

If you’ve read my book, and built some interoperable Web based systems, you might come to the same conclusion as I did. If that happens, I count my book as a success.

Node.js – Cost of IPC

2012-03-23T17:52:06Z

I ran a quick test to quantify the cost of IPC between workers and the master in a node.js cluster. The test code is below (which I borrowed from node’s docs). On receiving an HTTP request, each worker process sends a message to the master. I ran three variations – no messages, send one message, and send 5 messages. 1. Don’t send a message. 2. Send one message per request 3. Send five messages per request These are done on Ubuntu 11.04 on a quad-core Xeon box using node 0.6.12.