Andrew brought up something that’s disturbing, and I’ve seen elsewhere, which is that after seeing our presentation last year at Velocity, some folks decided that we somehow gave an endorsement to the idea of pushing your code whenever you want, and let the ‘ops guys’ deal with whatever comes as a result. Which isn’t at all what we suggested, and pretty much against the ideas of cooperation and communication between the dev and ops teams. I talk a bit about this in the podcast.

You have to prove that pushing whenever you want is an ok (safe, secure, etc.) thing to do. And the minute you can’t prove it, and you decide to continue that way….IMHO: you’re doing it wrong.

Problem is, it’s in a twisty-maze of dependencies. It’s running FUDforum 2.6.4RC1, on MySQL 3.23, on RedHat 9 (!). It needs to somehow get backed up, moved, and upgraded to latest FUDforum (3.0.0) and MySQL 5, on the new machine.

It’s not 100% straightforward, needs someone who’s done this before, and someone who isn’t me, because of the new job and all.

If you know someone who can help out, please email me where my email address is jallspaw which is located on a server whose domain name is yahoo.com.

Thanks!

UPDATE: I found a guy.  And he’s great with FUDForum. Excellent!  Thanks all those who emailed!

In what Andrew has called ‘boundary objects‘, deployment of new code has been a rallying point for the devops crowd, and I think that’s great. Deployment is definitely one of the places where the rubber meets the road. In some organizations, deployment of new code can be the single-most stressful and dividing parts of their work. People get fired or quit because of the emotional baggage that can come with an event that in the worst case, is nothing more than a planned outage disguised as progress and a followup finger-pointing session. Some groups have such dysfunction that they might as well just not even deploy the code. Just skip that part, head into a conference room, and fight bareknuckle. Toxic would be the nice way of describing those environments.

So it’s no wonder that a lot of the emphasis in this growing “devops” community is on deployment. Whether it’s providing confidence in changes with rigorous testing, deploying small changes often, dark launching, feature flags, or building a one-button deploy system – any effort to reduce the risk of change should be considered mandatory, IMHO.

But at the same time, deployment is only just a part of what really makes a great environment for development and operations to collaborate. Really. It’s not just about developers collaborating on deployment and releases. It’s about both teams understanding each other’s responsibilities after code is deployed to production, and collaborating along the areas of their expertise in a way that’s constructive.

Good Operations teams already write code, just not usually user-facing code. They spend a good deal of their time writing code to gather information from the infrastructure and act on it with short, medium, or long-term goals, usually aimed at performance and availability.

I’ll say that things like:

• metrics collection
• monitoring and associated thresholds
• load-feedback behavior
• instrumentation
• fault tolerance

should also be considered boundary objects between development and ops.

This is some of what I mean by that:

Metrics collection

I’ve said this before, but context is absolutely everything. Application-level or feature-level metrics is what gives the missing context to in-the-box resource usage like CPU, disk, memory, or network. At Flickr, the ops group maintains a number of different platforms for gathering metrics, like ganglia. To make it easy to add metrics, some of our backend applications will just write a temp file with key value pairs that we want to have squirted into ganglia. Like:

image_processed=30

image_processing_time=5

and ganglia’s gmetric cron job will pick that up every minute with the key as the metric name, and the value as, well, the value.

This means that all developers have to do is drop that file into an expected location and it will do the right thing. No tickets for making a new metric, no need for writing yet another script to gather a single metric, no need to understand the intricacies of whatever metrics collection system you have.

That’s an example of technical collaboration between the two groups. The missing piece is the cultural bits, which is the developer communicating their motivation behind getting these in-app metrics gathered and put on a graph. This gives the metric context, and might give ops some ideas on how they could use the metric for monitoring, capacity, or other purposes.

Monitoring

Involving development in designing your monitoring system can help provide a great perspective on failure modes. Peer code reviews are common in software development, so why shouldn’t monitors be reviewed? It’s still code, and it’s going to provide your humans (and maybe machines) with the data needed to fail gracefully, heal itself, or inform developers on what their constraints are when building new things. Your monitoring system is just like your code in that it should always be evolving, alongside your growth.

Remember all the raves about Google Analytics adding “intelligence” and alerts? Having some notion of thresholds isn’t just for people answering pages from nagios, it’s for everyone. How else can you gauge your expectations and guide future modifications to your code with respect to resource usage?

Load feedback behavior

Like a lot of smart web infrastructures, we’ve built an offline tasks system, which will asyncronously run jobs on our data that don’t have to be real-time. If you haven’t read Myles’ post on it, you really should. It’s a huge part of our strategy to avoid pretty common scalability pitfalls.

Anyway, these tasks, which can be relatively hard on the databases (which is one of the reasons why we do them asyncronously in the first place) have some built-in feedback mechanisms: they’ll check if there’s an unreasonably high number of concurrent MySQL connections, or if the database shard master-master pair doesn’t have both servers in production, or otherwise can detect that either what it’s trying to do on the database is too harsh at the moment. Whether it’s because of current live traffic being high, or a loss of redundancy, the offline task system will stop what it’s doing and re-queue it for later. This is a great (and safe) way of schmearing out heavy loads over a longer time period, reducing their risk.

Throw in some metrics collection about the size of those queues, and monitor alerts to do something for low or high-water mark thresholds, and then you’re cookin’ with gas.

Instrumentation

Through the magic of apache notes, developers can send extremely useful bits from within php code to the access and error logs. At Flickr, we’ve got some pretty simple notes set to help track things down when there are issues. For example. when I load the page for my photostream, the log line looks something like:

www394 123.456.789.012 5555 173663 [14/Dec/2009:04:08:21 +0000] “GET /photos/allspaw HTTP/1.1″ – 200 18233 “-” “Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1.3) Gecko/20090824 Firefox/3.5.3″ – -

where 5555 is my user id. Since php knows you’re logged in when you view a certain page, there’s no reason why we shouldn’t just log that in the request, so if there are any user-specific issues, it’s not a needle in a haystack.

Another example are API requests. We’ll log the api key making the call along with the authenticated user id, even in POST requests. Being able to trace a bullet through the entire request and response via logs is obviously handy. Putting user ids, API methods, and API key specific info into log lines is hugely helpful when troubleshooting issues, especially if you’re running one of the most popular APIs on the web.

Fault Tolerance

Ross blogged about how we do feature flipping last week. He goes over how important (and awesome) this is to our development process, but another one of the advantages of this approach is how it affects operations.

This is an example of development taking an active role in not only deployment, but the time and effort to operationalize features and pieces of code so that in cases of degradation or failure, these individual pieces can be forced to fail gracefully. Our talk at Velocity last year went over some of this, but it’s still one of the reasons why we can push code thousands of times a year and still have an extremely low MTTR whenever there’s an issue.

New code causing degradation? There’s an app for that! (it’s called a feature flag)

Anyway, my point is that deployment is only a small part of how development and operations should collaborate and communicate. In fact, dev and ops is only the most obvious starting point for getting along and working together on problems.

Product and community management also have important boundary objects with operations as well, but that’s for another blog post.

I do, of course, think this is changing as of recently. The punch line is that I obviously did take the job, despite my misgivings about not being surrounded by people who are constantly thinking about my industry. One of the reasons I got over not being in the ‘epicenter’ is that Fred Wilson and Albert Wenger did an insanely good job at convincing me it was a good idea. Another reason is that I think Etsy is basically a Bay Area company that just happens to be in Brooklyn. I mean that as a compliment.

So while I always had some inkling of what ‘epicenter of the web’ means, I was never really sure how that could be measured. Indeed.com has indirectly measured it by the # of job listings.  O’Reilly did something similar for the # of startup jobs in 2006.

Number of jobs is interesting, but I thought it might be fun to measure it by locations of headquarters as seen through the lens of monthly unique users. So, I took the Quantcast “Top 100″ sites, found the latitude and longitude of the headquarters of each site via Crunchbase’s API, as well as other bits around the web, and Aaron helped out with the excellent Modest Maps to make this:

Quantcast Top 100 plotted on U.S. Map, radius = monthly uniques

Like I said, this doesn’t change my thoughts about the new job, or what I think ‘epicenter of the web’ means. But, still interesting, dontcha think?

UPDATE: Here’s a link to the raw data: http://spreadsheets.google.com/pub?key=tLwD1C5mghn9U3XJj_yqyjw&output=html

If there’s anything wrong, lemme know.

When Stew and Cat asked me to join Flickr in January of 2005, I felt like it was time to go and do something different, so I said yes.

Five years (and four billion photos) later, it’s again time to go and do something different. It’s hard for me to describe what a blast this has been. Our goal was to kick ass, and I think we did. Flickr has served as the  backdrop of some of the largest changes in my life, and the work I’ve done there is essentially tied to those events in my memory.

During my time here at Flickr, I:

• moved house
• saw the company get bought by Yahoo!, and worked out that whole transition thing
• got married
• had a daughter
• co-invented a pretty well-received office sport
• wrote a book
• had a son

In addition to building, scaling, evolving, and generally being as loud and fast as we could possibly be with the original Ludicorp team, I had the absolute privilege to hire and work in the trenches with some of the greatest people on the web. I also had the chance to work with some of the smartest people at Yahoo, who I’ll continue to have relationships with even after I leave. Yahoo has treated me well, and I’ve learned more here than I have at any other company.

The reason I stayed here for five years wasn’t for the accolades (or the vesting). It was because I worked with people who care about building something that people care about.

This also happens to be the same reason why I chose my next step: Etsy. They care, and it shows.

I still have a little more time here at Flickr to rock a bit more, but I’m excited to work with my friend Chad again on something that matters. I’ll be running the Ops group there, where they’ve already got superstars.

Chad wrote some more about it here.

Let me start with this: I don’t think I can overstate how right-on this paper is, with respect to the challenges, solutions, observations, and concerns involved with operating a medium to large web infrastructure. I found this via @benjaminblack, and I agree with him 100%: this should be considered required reading for anyone in our industry. I’m not sure if Cook ever thought that his paper would apply to web infrastructure, but I think it can and does. Please take 30 minutes right now and read it.

There are a number of salient points in the paper that I’d like to comment on. Again, this is through the lens of failures of complex systems as it pertains to web operations:

7) Post-accident attribution accident to a ‘root cause’ is fundamentally wrong.

I’m going to guess that this portion may be viewed as controversial in the prevailing webops wisdom, where post-mortems are for sure necessary, but whose content may or may not be effective in preventing similar types of failure. I do value the process of a post-mortem, because I think the human element of understanding complex failures is important and doing whatever you can to put in place safety is good, modulo what is said in section #16 of the paper. I believe that even a rudimentary process of “5 Whys” has value. But at the same time, I also think that there is something in the spirit of this paragraph, which is that there is a danger in standing behind a single underlying cause when there are systemic failures involved. Doing this can lead to the false belief that you’ve got this mode covered, you’ve found the silver bullet that made the whole mountain crumble, and jeez what a relief because that will never bite us again.

14) Change introduces new forms of failure.

I totally agree with this point. However, I often see this as a rallying point for operations teams to say “No!” to change, when instead they should be working alongside development (and product owners) with a goal of reducing the risk of failure associated with each change. I do not believe that ‘release early, release often’ in and of itself can reduce that risk. I believe that the real (and only) way to do this is both technical and cultural. But I’ve spoken about this before.

16) Safety is a characteristic of systems and not of their components

Emphasis on “Safety cannot be purchased or manufactured; it is not a feature that is separate from the other components of the system.” Real safety comes from smart people doing smart things to the entire shebang, not the individual guts.

and I think the point I love the most, with all of my heart:

18) Failure free operations require experience with failure.

Fear is a strong emotion. I believe it can be used as a strong motivator for ensuring safety in the face of constant change, instead of a reason to push back on the very idea of change. Embrace fear of outages and degradation. Use it to guide your architecture, your code, your infrastructure. So lean into it.

There are a lot of great points in the paper, and I could go on, but you get the idea.

What:

…did we do before (historical trending, etc)
…is going on right now? (troubleshooting, health, etc.)
…is coming down the road (capacity planning, new feature adoption, etc.)
…can we do to make things better (business intelligence, user-behavior, etc.)

All of which, of course, should be considered mandatory in order to help your business increase its awesome. Yay metrics!

Some time ago, Matthias wrote great a blog post about some of the metrics that can reasonably profile the effectiveness of web operations, taken from the ITIL primer, VisibleOps.

In my opinion, there’s nothing on that list of things that isn’t valuable, as long as the cost of gathering those metrics isn’t too behaviorally, technically, or organizationally expensive. The topics included in that list of metrics and the context they live in is fodder for many, many blog posts.

But in the category of historical trending, I’m more and more fascinated by gathering what I’ll call “meta-metrics”, which is data about how you respond to the changes your system is experiencing.

One of the best examples of this is gathering information about operational disruptions. Collecting information about how many times your on-call rotation was alerted/paged/woken-up, during what times, and for what service(s) can be enlightening to say the least.  We’ve been tracking the volume of alerts a lot closer recently, and even with the level of automation we’ve got at Flickr, it’s still something you have to keep on top of, especially if you’re always finding new things to measure and alert on.

Now ideally, you have an alerting system that only communicates conditions that need resolvable action by a human. Which means every alert is critically important, and you’re not ignoring or dismissing any pages for any reasons that sound like “oh, that’s ok, that cluster always does that…it’ll clear up, I’ll just acknowledge the page so I can shut up nagios.” In other words, our goal is to have a zero-noise alerting system. Which means that all alerts are actionable, not ignorable, and require a human to troubleshoot or fix. Over time, you push as much of this work as you can to the robots. In the meantime, save humans for the yet-to-be-automated work, or the stuff that isn’t easily captured by robots.

Why is this important to us? I may be stating the obvious, but it’s because interrupting humans with alerts that don’t require action has a mental and physical context switching cost (especially if the guy on-call was sleeping), and it increases the likelihood of missing a truly critical page in a slew of non-critical ones.

Of course in the reality of evolving and growing web applications, even if we could reach a 100% noise-free alerting system, it’s impossible to sustain for any extended period of time, because your application, usage, and failure modes are constantly changing. So in the meantime, knowing how your alerts affect the team is a worthwhile thing to do for us. In fact, I think it’s so important that it’s worth collecting and displaying next to the rest of your metrics, and exposing these metrics to the entire dev and ops groups.

Something like this: (made-up numbers)

Tracking Critical Alerts

Gathering up info about these alerts should give us a better perspective on where we can improve. So, things like:

• How many critical alerts are sent on a daily/hourly/weekly basis?
• What does a time histogram of the alerts look like? Do you get more or less alerts during nighttime or non-peak hours?
• How much (if any) correlation is there between critical alerts and:

- code deploys?
- software upgrades?
- feature launches?
- open API abuse?

• What does a breakdown of the alerts look like, in terms of: host type, service type, and frequency of each in a given time period?

and maybe the most important ones:

• How many of those alerts aren’t actually critical or demand human attention?
• How many of them always self-recover?
• How many (and which) don’t matter in their role context (like, a single node in a load-balanced cluster) and could be turned into an aggregate check?

We’ve built our own stuff to track and analyze these things. My question to the community is: I’m not aware of any open-source tool that is dedicated to analyzing these metrics. Do they exist? Nagios obviously has host/hostgroup/cluster warning and critical histories, and those can be crunched to find critical alert statistics, but I’m not aware of any comprehensive crunching. Of course, until I find one, we’re just building our own.

Thoughts, lazyweb?

High availability

Having redundant infrastructure is WebOps 101. For my kids’ most prized possessions, their sleeping  ‘loveys’ there is no reason to have a SPOF, under any circumstances. We have at least 4 backups for each on any trip that we go on, as well as a couple of trusted stuffed animals who might meet unfortunate fates.

Capacity planning

This applies to both disposable diapers (a.k.a. consumable capacity) and episodes of the few TV shows we allow them to watch, on the Tivo. My daughter, at 3 and a half, knows every detail from every of the 49 episodes of The Backyardigans. Having some of them on ipods and iphones can make a 6 hour drive to L.A. feel like 4, not 12.

Documentation

Since I’m already used to writing down observations and techniques learned ‘in the field’, then I was totally prepared:

Allspaw Baby Soothing Method, v1

and in case I ever forgot what my most successful swaddling method was:

Architecture and design

It’s unfortunate that I was so sleep-deprived that I never got a photo of the RadioShack remote-control truck that I turned into a cam-driven Moses basket automatic rocker mechanism. But you understand what I’m talking about.

There is one other thing that I learned from working at Flickr which turned out to be useful new parent advice: expect the unexpected, and never rely on past behaviors as an indication of what can happen in the future. They’re kids, not applications.

One of the cooler papers is “Automatic exploration of datacenter performance regimes” in which the smart folks over at the RAD Lab at UCB tackle the idea of:

1. Gathering up real usage metrics in production
2. Taking that data to feed a resource allocation (”auto-scaling”) controller

The bits about coming up with an exploration policy is where the juicy stuff comes in, building in safety factors driven by external SLAs. You should read the whole thing to see how thoughtful their method was, which includes taking into account effects such as cold ramping, which you almost never see accounted for in simulated situations.  Rock on, RAD Lab: this is the stuff that brings the academia smarts to the real world. Kudos.

FYI: I’m not just saying the paper is cool because they cite my book as a resource in it.