Operation Bootstrap

DevopsDays 2013 - we are avoiding culture, why?

2013-05-02T21:10:00-05:00

I just got back from Devops Days Austin. It was a really good conference. I think I enjoyed the speakers and Open Spaces at this event more than I did at Devops Days Mountain View last year. Huge props to the organizers and speakers for putting together such a great set of topics.

One thing irked me a bit though, I heard a number of comments about how there was too much talk about culture.

I am confused - but I think I understand.

I attended one Open Space session called “Culture Hacks” which ended up not being so much about hacks as a plea for help. A number of folks expressed concerns about being in a difficult situation where they associate their problems with organizational cultural problems and they were looking for ideas on how to make things better. The suggestions that folks raised, myself included, sounded a lot like typical lean/agile tools - retrospectives, stand-ups, putting developers on-call, communicating metrics, providing incentives, hackathons. I think these are all good suggestions, the problem is that they only go so far. The unfortunate reality is that old saying - you can lead a horse to water, but you can’t make him drink. These are all tools but they don’t solve a problem in an organization that doesn’t want to change.

I think the topic of culture is pretty big and scary for many folks. It’s also very subjective, as John Willis raised in his talk - the culture in which a group of crooks thrive is very different than the culture in which a group of hippies would thrive (John didn’t use Hippies in his example). For each group though, there is a specific culture that allows those individuals to reach their goals in an optimal way. Is one culture right or wrong? Functional or dysfunctional? I think the answer is that if the culture is aligned with the team and makes them perform in an optimal way then it’s good - but it’s good for that group, not for everyone.

As such, if you haven’t defined an objective for your culture, if you’ve hired a mish-mash of people with different objectives and principles then you really aren’t going to find a culture that makes everyone achieve their goals. You can make it better for some, but it’ll likely not be better for others. Getting the folks off the bus who do not align with your desired culture is an important part of a change like this and is not something that an Ops person in one group can do. Here, I think, lies one of the main barriers to DevOps being about culture change - it’s driven by Ops people.

Ops working with Dev is great, we can all do that, but we are individuals who can only set an example and hope the organization follows suit. If they don’t - you can vote with your feet or suck it up, I’m not sure there are many other options. Setting an example often manifests as Ops folks trying get closer to Dev - the most natural way to do this is to write code, help with release engineering, enable developers to gain access to monitoring, logs, config management, etc. All the tools that we say aren’t actually what Devops is about… These seem to me to be a manifestation of ops folks doing what they can do get closer to Dev, that’s all.

Of course it isn’t culture, company culture is bigger than this, but it can change a small part of an organization & help set an example for the broader organization. Sometimes it can have a bigger impact - I love this one by John Allspaw as an example. Still, the change was isolated.

I recently read & really liked this TechCrunch article because I feel like it hits on an important point. If I work at a company where the CEO, CTO, COO, VP of Engineering or a variety of other high level positions push a culture that I don’t like - my chances of changing that are small. Further still, there is already a culture which is somewhat defined by the team you’ve hired. Unless you’ve worked very hard to hire for a specific person & team fit then no amount of effort is going to change the organizational culture.

So what is my point with all of this? While Dev & Ops collaboration is important to having a healthy development process - it is so easily undermined by problems with the larger organization. I don’t think that means you do nothing - I just think it helps clarify why talking about culture is hard. I also think having more actionable steps, more examples to replicate would be helpful. Right now the examples we have are of companies who have a good culture throughout - but what about companies who can’t do that? How do I fix my piece? How do I get my 10 person startup pointed in the right direction? How can I build a functional bubble inside a dysfunctional behemoth? What can I do to make my small corner of the world suck less?

There aren’t enough good examples for these questions - and this is why we need to talk about it. Talking about the little bits and pieces that work for you helps - so don’t avoid talking about it because you don’t have all the answers - share what you can. In that Open Space I attended, plenty of folks had little ideas about things that worked for them - they didn’t have a complete solution, but they had some ideas.

I shared my thoughts about how this may look when you start from the ground up and we’re implementing some of these ideas at my current company. Are they all the right things for this company? Probably not - the team will decide. Some of the things that worked at a past company wont work here - does that mean we are doomed? I don’t think it does.

Culture is very personal - but so are a lot of things that we have some established patterns for. We are homing in on some ideas that work - we need more ideas - and we need to talk about them more. I hope this comes up more at other Devops Days events in the future. It’s an important topic.

Good & Bad patterns in Development and Operations

2013-01-14T16:36:00-06:00

As part of my role at a new company I’ve been asked to provide feedback about structuring Dev & Ops as well as what sorts of things work and don’t. I certainly don’t claim to have all the answers, but I’ve seen some very functional and some very dysfunctional organizations. I’ve spent a fair amount of time thinking about what works & why.

Below is a cleaned up version of a message I sent to our CEO who asked for my thoughts on what does and doesn’t work. This was intended as scaffolding for further discussion so I didn’t go into deep details. If you want more details on any particular area just throw some comments out there.

I realize not all these issues are black & white to many folks - there are gray areas. My goal with this message was to drive conversation.

I figure this is probably review to many folks, but maybe it’ll help someone.

First, there are some very simple goals that all these bullets drive toward & they’re somewhat exclusive to SaaS companies:

Customers should continuously receive value from Developers as code is incrementally pushed out
Developers should get early feedback from customers on changes by enabling features for customers to test
We can address problems for customers very quickly - often in a matter of hours
We can inspect and understand customer behavior very deeply, gathering exceptional detail about how they use the service.
We can swap out components & substantially change the underlying software without the customer knowing (if we do it right)
We can measure how happy customers are with changes as we make them based on behavior & feedback

The lists below are what I feel make that possible (Good) and what inhibit it (Bad)

Culture & Communication

Good:

Stand ups.
Retrospectives
Small, self-formed teams (Let folks work on their area of passion)
Use Information Radiators whenever possible (Kanban boards, stats on big monitors, etc)
Decisions by teams, Leaders facilitate consensus
Discovering what doesn’t work is part of finding the right solution, not something to fear.
Hackathons allow Developers to do things they are passionate about
Hire for personality & team fit first, technical ability second
Data driven decisions, strive to have facts to back up decisions.
Make the right behavior the easiest thing to do - build a low resistance path to doing the right thing.

Bad:

Top down decision making
Strict role assignments & Silos
Fear of not getting it right the first time
Hiring for technical ability thinking team fit will come later
Creating process out of fear that makes it difficult to do the right thing.

Eliminate Manual Processes

Good:

Continuous Deployment / Delivery
Fully automated testing
Test Driven Development
Fully automated system monitoring, configuration & provisioning
Separate Deploy & Release (Feature toggles)
Deploy from master, do not branch (Forces particular behaviors)

Bad:

Manual testing by a QA Team - sometimes it’s necessary, but should be avoided
Deploying off a branch, slows things down & allows for other bad behaviors
Writing tests after writing code, code isn’t written with testing in mind
Developers relying on other teams to perform tasks that could be automated.
Processes that are the result of fear rather than necessary business process.

If it moves, measure it

Good:

Collect high resolution metrics about everything you possibly can
Developers can add new metrics by pushing new code, do not rely on additional configuration by other teams.
Graphs & metrics can be seen by anyone - Developers should rely on these.
There should be individuals or teams who are passionate about data visualization & analysis.
Dev teams rely on these metrics to make decisions, help identify what metrics are important
Developers watch metrics after pushing new code, watch for trend changes (Devs take responsibility for availability)

Bad:

Operations has to configure new metrics after developers have added support for them (Manual)
Operations monitors metrics & asks Dev teams when they think there’s a problem
Developers don’t look at metrics unless something is brought to their attention
Code doesn’t expose metrics until someone else asks for it

And here is the long version of all of that…

#1 Culture & Communication

Above all else I consider these most important. I think most problems in other areas of the business can be overcome if you do well in these areas. Rally has been, by far, the best example of a very successful model that I’ve seen in this area. They aren’t unique - there are other companies with similar models & similar successes.

Main points

Stand ups. By far the most effective tool for keeping everyone in touch. As teams grow you have to break them apart, so you have a 2nd standup where teams can bring cross-team items to share.
Projects are tackled by relatively small, typically self-formed teams. Get individuals who are interested in working in an area together & they feed on each others passion.
Perform retrospectives. This gives individuals & small groups the ability to voice concerns in a way that fosters resolution. There’s an art to facilitating this but it works well when done right. It also allows recognition of things that are done well.
Use open information radiators - it should be easy to see what’s going on by looking at status somewhere vs. having to ask for status, go to meetings, etc. Kanban boards are great for this.
Leaders exist to facilitate and help drive consensus but decisions are largely made by teams, not leaders. This makes being a leader harder, but it makes the teams more empowered.
Accept that things may not work & the team and company will adjust when things do not work. This makes it easy to try new things & easy for people to vocalize when they think it isn’t working. If it’s hard to change process then people are more resistant to try new things. This goes back to retrospectives for keeping things in check. Also important in this are “spikes” or time boxed efforts explicitly designed to explore possibilities.
Give developers time to pursue their own projects for the company. Many awesome features have come out of Hackathons where developers spent their own time to build something they were passionate about.
Hire for personality fit first. I have seen many awesome people find a special niche in a company because they grew into a role that you couldn’t hire for - but what made that possible was that they worked well with the team as an individual. Hiring for technical skill also means you lose that skill when that person leaves, I would prefer to have cross-functional teams.
Data driven decisions. This helps keep emotion and “I think xyz” out of the discussion & focuses on the data we do and do not have. If we don’t have data we either get more or acknowledge we may not be making the right decision but we’re going to move forward.
Make the right thing the easiest thing. I’ve seen too many companies put process out there that makes the “right thing” really difficult, so it gets bypassed. The right thing should be an express train to done - very little resistance and very easy to do. It’s when you start wanting to do things differently that it should become harder, more painful.

Also, everyone owns the quality of the service. This includes availability, performance, user experience, cost to deliver, etc. At my last company, there was exceptional collaboration between Operations, Engineering and Product (and across engineering teams) on all aspects of the service and there was a strong culture of shared ownership & very little finger pointing.

If you want more details on this specific to Rally I wrote a blog post with some more info: Blog Post

#2 Obsessively eliminate manual process - let computers do what they are good at.

This is so much easier to do up front. There should be as little manual process as possible standing between a developer adding value for customers (writing code) and that code getting into production. There may be business process that controls when that feature is enabled for customers - but the act of deploying & testing that code should not be blocked by manual process. I refer to this as separating “Deploy” from “Release” - those are two very different things.

Testing should only be manual to invalidate assumptions, validating assumptions should be automatic When we assume that if x is true then y will occur, there should be a test to validate that this is true. Testers should not manually validate these sorts of things unless there is just no way to automate them (rare). Testers are valuable to invalidate assumptions. Testers should be looking at the assumptions made by Developers and helping identify those assumptions that may not always be correct.

Too many organizations rely on manual testing because it’s “easier”, but it has some serious drawbacks:

You can only change your system as fast as your team can manually test it - which is very slow.
Your testing is done by humans who make mistakes and don’t behave predictably so you get inaccurate results.
The # of tests will only grow over time, requiring either more humans or more time, or both. It doesn’t scale.

Over time the software quality gets lower, takes longer to test, and the test results become less reliable. This is a death spiral for many companies who eventually find it very hard to make changes due to fear & low confidence in testing.

Avoiding this requires developers spend more time up front writing automated tests. This means developers might spend 60-70% of their time developing tests vs. writing code - this is the cost of doing business if you want to produce high quality software.

That may seem excessive, but the tradeoffs are significant:

Much higher code quality which stays high (those tests are always run, so re-introduced bugs (regressions) get caught)
Faster developer on boarding, the tests describe how the code should behave and act as documentation.
Refactoring code becomes easier because you know the tests describe what it should do.
Each commit to the codebase is fully tested, allowing nearly immediate deployment to production if done right.
Problems that make it into product feed back into more tests & continually improve code quality.

Much of the time developing tests is spent thinking about how to solve the problem, but you are also writing code with the intent of making it testable. Code is often written differently when the developer knows tests need to pass vs. someone manually testing it. It’s much harder to come along later and write tests for existing code.

You will hear me talk about Continuous Deployment & Continuous Integration - I feel these practices are extremely important to driving the above “good” behaviors. If you strive for Continuous Deployment then everything else falls into place without much disagreement because it has to be that way. This has a lot of benefits beyond what’s listed above:

Value can be delivered to customers in days or hours instead of weeks or months
Developers can get immediate feedback on their change in production
New features can be tuned & tweaked while they are fresh in a developers mind
You can focus on making it fast to resolve defects, no matter how predictable they are, rather than trying to predict all the ways things might go wrong.
Most of the tools and behaviors that enable Continuous Deployment scale to very large teams & very frequent deployments. Amazon is a prime example of this, deploying something, somewhere, about every 11 seconds. Many companies that are in the 30-100 engineer size talk about deploying tens of times per day.
This also impacts how you hire QA/Testers. This is a longer discussion, but you want to hire folks who can help during the test planning phase & can help Developers write better tests. Ideally your testers are also developers & work in a way that’s similar to Operations, helping your Developers to be better at their jobs.

#3 If it moves, measure it

I mentioned above, two big advantages a SaaS organization has are the amount it can learn about how customers use the product & the ability to change things rapidly. Both of these require obsessive measurement of everything that is going on so that you know if things are better or worse. Some of these metrics are about user behavior & experience to understand how the service is being used. Other metrics are about system performance & behavior.

The ability to expose some % of your customer base to a new feature & measure their feedback to that is huge. Plenty of companies have perfected the art of A/B testing but at the heart of it is the ability to measure behavior. Similar to testing, the software has to be built in a way which allows this behavior to be measured.

System performance similarly requires a lot of instrumentation to identify changes in trends, to identify problems areas in the application & to verify when changes actually improve the situation.

I’ve been at too many companies where they simply had no idea how the system was performing today compared to last week to understand if things were better or worse. At my last company I saw a much more mature approach to this measurement which worked pretty well, but it required investment. They had two people fully dedicated to performance analysis & customer behavior analysis.

Continuous Deployment: Are you afraid it might work?

2013-01-14T07:47:00-06:00

I’ve been wondering for a few years now, why it’s so hard to get companies to prioritize the work that I feel is important. I mean, I’m telling you how to do it and you aren’t listening - don’t you want to build quality software?

Would you listen to that argument? I wouldn’t. Everybody has an opinion about how to do things, what makes one better than another?

I think you should listen to me, but that’s irrelevant

I’m on my 4th SaaS company at this point. I’m starting early this time and hoping to steer things in the right direction. I feel like I’ve observed some good and some bad and some really ugly at this point and I have a pretty good idea of what patterns and anti-patterns are important. The problem hasn’t changed though, just because I feel they are important doesn’t make them a priority for the company.

When I go to work for a new company a very important question to answer is whether the company is ready and willing to implement all the cultural and technical requirements for Continuous Deployment. I’ve at least figured out that, from my position, it’s exceptionally hard (so far impossible for me) to convince a company they want to do this - they have to already want to. I know how to implement, I know how to enact change, but I need support that has to already exist.

I focus on Continuous Deployment not because it’s a technical solution that you just plug in and go. I focus on it because it drives conversation around all the other areas where organizations should improve. For each improvement you make working toward Continuous Deployment, you make your development process better and your software better. These aren’t things that only provide benefit once you are doing Continuous Deployment - but when you’ve done them all it becomes a fairly easy decision to deploy continuously.

I’m early into my latest venture but already the attention is there, the interest in doing this right. I’m being asked for my thoughts on what we need to prioritize to move toward Continuous Deployment, what do we need to focus on early so that it’s easier later on. I’m also being asked to educate folks on what is important and why, what have I watched work and what have I watched fail. What mistakes can we avoid and what mistakes are we just going to have to make on our own?

Oh, you were hoping not to make mistakes? Good luck with that. The best I can hope for is to make my own mistakes.

Dude, I would never do that…

My first flight on an airplane, in my life, was a skydiving trip. When the instructors discovered this as we ascended toward altitude they said “Well this is perfect, next time you fly you can tell the person next to you that you’ve flown before, but you’ve never landed”. My risk tolerance may differ slightly from others. I like to rock climb as well, plenty of people won’t do that. The thing is, both of these sports involve a very risky activity offset by copious amounts of safety. Still, when you watch the girl up on the cliff hanging by a limb all you can think of is “what if she falls?”. The answer is, she’ll get a little bruised up maybe, but if she’s doing things right she’ll keep right on climbing.

When you’ve been climbing a bit, when you understand the safety mechanisms, you pay much more attention to the climb, to the technique, to each movement. You know that the climber is probably safe, because you know what keeps them safe. You can take more risk with each movement knowing that any single mistake will only set you back so far.

These activities, like Continuous Deployment, look more risky from the outside than they are. If you don’t take the time to understand all the safety mechanisms then you can’t accurately evaluate the risk. For a company who pushes software every few weeks to consider pushing every commit without substantial other changes would be insane. Just like I would never go rock climbing without the right equipment (I’m allergic to free soloing - sorry). The act of Continuous Deployment is a realization of a ton of other effort - and all that effort has to be prioritized before you can ever get on the rock face.

Reality check - you are deploying more often than you think

Lets say you deploy every week - I’m being generous here, but lets just pretend. So you deploy on Thursday during the day because you have an awesome deploy process and you know it’s better to spot problems when everyone is in the office. You spot a problem, what do you do? I’m guessing your deploy was from a branch, so you just fix that branch & deploy. Then you merge the fix into master.

Friday comes along, hey there’s another critical issue. Fix, branch, deploy. Lather, rinse repeat. Meanwhile, depending on how involved the fix is and what other stuff you have going on, you’ve got a bunch of merging to get right and the closer to your next branch you get, the more of a problem this becomes. How about a fix on Wednesday before the next deploy? I’m guessing you’ve already cut the next branch, so now you apply the fix to 3 branches (last week, this week, master).

All this deploying and merging and branching, it’s all work. The problem is - it’s not automated work, it’s work asking for mistakes to be made. It’s risk. Where are your safety mechanisms? Are they your manual testers? Your automated test suite? If those automated tests aren’t good enough to test each commit before it goes to production, why are they good enough to test each weeks deploy? Because you do manual testing?

This all sounds risky to me, but for some reason it sounds less risky than Continuous Deployment to some. I think this can only be because of a lack of understanding around the safety mechanisms, the pre-requisites. The proof is in the pudding though, and if you still produce shitty software when doing Continuous Deployment because you write bad tests and don’t do retrospectives and don’t prioritize the important work of making the system work right - then you’re sunk either way.

There are some companies that are probably better off deploying every 8 weeks.

Wrapping up - why so much focus on Continuous Deployment?

The practices that surround Continuous Deployment/Delivery substantially reduce risk - things like Feature Toggles, automated testing, automated deployments, deploying off master, retrospectives, monitoring, accountability, access, ownership, reduced MTTR, and the list goes on. These all add up to make a software development and deployment environment so safe, anyone can commit code - if it doesn’t work it will not make it to production.

But, things will still break. In my experience you have to break things in very subtle ways for code to get into production & as time goes on and you build better monitoring, even those issues should be detected fast & resolved fast.

It can take a while to reach the end goal, but you’ve got to start somewhere. However, even if you never actually practice Continuous Deployment, all of these practices will produce better software and probably happier developers.

DevOps is Culture - so is a lot of other stuff...

2012-06-29T22:20:52-05:00

I hung out in an excellent discussion at DevopsDays driven by Spike Morelli around culture. The premise was that DevOps started as an idea around culture - around behavior patterns that lead to better software. Somewhere along the way our industry shifted this discussion into a tools discussion & now the amount of noise out there about “DevOps tools” is magnitudes higher than any discussion about the real reason DevOps exists - to shift culture.

I looked up the definition of “culture” - here are a few definitions:

the quality in a person or society that arises from a concern for what is regarded as excellent in arts, letters, manners,scholarly pursuits, etc.
that which is excellent in the arts, manners, etc.
the behaviors and beliefs characteristic of a particular social, ethnic, or age group: the youth culture; the drug culture.

Note that culture is the manifestation of intellectual achievement. It’s the evidence and result of achievement. I think the 3rd definition is most appropriate for DevOps - what are the behaviors that are characteristic of a well integrated Development & Operations organization?

The challenge, the discussion, was how we can re-balance the scales and get the word out that this is actually about culture and that tools happen as a result of culture, not the other way around. This post begins my contribution to that effort.

The question was asked - do we all agree that culture is the most important thing when it comes to creating a successful business? The short answer is “yes”. If you wanted to hear all the if/and/but/what-if/etc discussions, you should have come to Devopsdays. For the sake of this blog post - culture is the most important factor. If you want case studies and analysis that proves that culture matters - read Jim Collins Good to Great.

My present company has a really excellent culture of Developer / Ops cooperation and collaboration. I wasn’t there when it wasn’t that way (if ever) and so I can’t tell a story about how to change your organization. What I can tell you is what a healthy and thriving Dev/Ops practicing organization looks like and what I think some of the key factors in that success are. I see this as two components - there are fundamental core values that enable and support the culture, and then there are tactical things that are done to make the culture work for us. I’d like to talk about both. The culture is the result of these actions and ideas put into practice.

Background

I work for a company with a well defined set of core values. Those values set forth parameters under which the culture exists. Here’s what they are:

These values are public and they matter - they matter a lot.

These might sound hokey to you - but every single one of them is held high at the company & strongly defended. Defending a list of values like this is hard sometimes. When someone doesn’t show respect to others, how do you uphold that core value? When someone’s idea of “work life balance” is different than another person, how do you support both of them? When creating your own reality means you don’t want to work for Rally anymore - what do you do?

I’m proud to say that in Rally’s case - they are generally true to the core values. Putting “Create your own reality” on a list of core values doesn’t create culture - what creates culture is having repeated examples where individuals have followed their passion & the company has supported them. This support doesn’t just mean they have permission, it means the company uses whatever resources it can to help. Sometimes this means using your resources to help someone find another job. Sometimes this means helping them get an education they can use at another company. Usually though, it means getting them into a role where they can do their best work. Whatever the case - Rally’s culture is to always be true to that core value and do whatever they can to support an employee in creating their own reality.

This is repeated for all of the core values. By being explicit & public about these values they set the stage for what an employee can expect from Rally as a workplace. But there’s more to it - you have to make sure these core values are upheld and you have to make sure they thrive - and this is where some of the tactical parts come in.

What are the tactical things?

Collaboration - at Rally collaboration is a requirement. Development is done almost exclusively in pairs, planning is done as groups, retrospectives are done regularly and the actions from those retrospectives are announced publicly and followed up on. Architecture decisions are reviewed by a group comprised of Developers, Operations and Product.
Self Formed Teams - teams are largely formed of individuals who have an interest in that teams work. When we need a task force, an email will go out to the organization looking for people interested in participating & those teams self-form. This also gives anyone in the company the ability to participate in areas of the business they may never otherwise get exposure to.
Servant Leadership - Leaders at Rally often do very similar work to everyone else - they just have the additional responsibility of enabling their teams. Decisions about how to do things don’t often come from managers, they come from the teams.
Data Driven Decisions - Not strictly associated with a core value, I think this is one of the most important aspects of the Rally culture. There is an expectation that we establish evidence that a decision is correct. Sometimes this evidence is established before any dev work is done but sometimes this data comes from dark launching a new service or testing out some new piece of software. Either way, it’s understood that the job isn’t really done until you have data to support why a particular decision is right & have talked to the broader group about it.

There are plenty of other things here and there but you get the general idea. We talk a lot & tell each other what we’re doing, we enlist passionate individuals in areas they have interest, we embrace & seek out change and we empower individuals to drive change by working with others.

So what? What does that have to do with Devops?

Everything

2.5 years ago the company had some performance & stability problems. Technical debt had caught up with them and the only real way to fix the problem was to completely change the way the company did development & prioritized their work. The good news is that they did it, but it was made possible by the fact that individuals were empowered to drive that change. Almost overnight, two teams were formed to focus on architectural issues. A council was formed to prioritize architectural work. The things we all complain about never being able to prioritize became a priority and remain a priority to a degree I’ve never experienced before at other companies. Prioritizing this work is defended and advocated by the development teams - something only possible because of the collaborative environment in which we operate.

I have been personally involved in two services that literally started out as a skeleton of an app when they went into production. The goal was to lay the groundwork to allow fast production deployments, get monitoring in place & enable visibility while the system was being developed. This was all done because the developers understand the value of these things, but they don’t know exactly how to build it - they need Ops help. Having tight Ops and Dev collaboration on these projects has made them examples of what works in our organization. These projects become examples for other teams in the company and they push the envelope on new tech. These two projects have:

Implemented a completely new monitoring framework that allows developers to add any metric they want to the system
Implemented Continuous deployment
Established an example of how and why to Dark Launch a service

I’m sure the list will continue to go on… it’s fantastic stuff.

The Rub - culture isn’t much of anything without people who embrace it.

Along with a responsibility for pushing change from the bottom up in Rally comes responsibility for defending culture - or changing it. This means that when you hire people, they have to align with your core values - they have to be willing to defend that culture or the company as a whole needs to shift culture. All those core values and tactical things will not maintain a culture that the team members do not support. Rally’s culture is what it is because everyone takes it seriously and that includes taking it seriously when there’s a problem that needs fixing.

This has happened. There are core values that used to be on that list above but they aren’t anymore. At one point or another things changed and those core values were eroding at other core values. This takes time to surface, it takes time to collect data to show it’s true, but when the teams start to observe this trend they have to take action. This isn’t the job of management alone - this is the job of every member of the company. When the voice begins to develop asking for change - you need a culture that allows that change to take place and for everyone to agree on the new shape things take.

That said, it also isn’t possible if management doesn’t support those same core values. Management has the same responsibility to take those core values seriously.

DevOps is our little corner of a much bigger idea

There’s a problem that we’re trying to fix - we’re trying to improve the happiness of people, the quality of software, and the general health of our industry. Our industry is totally healthy when you look at the bottom line, but we’re looking for something more. We want a happy and healthy development organization (including Ops, because Ops is part of the Development organization), but we also want our other teams to be part of that. As Ops folks and Developers, we can clean up our side of the street - we can do better. We seek to set an example for the rest of the organization.

For culture to really improve in companies it has to go beyond Dev and Ops into Executives, Product, Support, Marketing, Sales and everyone else. You ALL own quality by building a healthy substrate (culture) on top of which all else evolves.

But in the end it’s about culture. It’s really only about culture for now - because when you get culture right the other problems are easy to solve.

Congratulations to those of you who read this far - shoot me a note and let me know you read this far because you probably share the same passion about this that I do. Also - putting up blog posts from 32,000 feet is awesome - thanks Southwest.

Rate my skills: no

2012-06-20T21:59:34-05:00

I’ve been asked in the past to rate my skills when chatting with a recruiter about a position, it goes something like this:

“Can you rate your skills in the following areas on a scale of 1 to 10:
Linux
Perl
SAN Storage
Networking
etc”

My response is generally, “no and goodbye”. This isn’t out of arrogance, but there’s just no point in moving forward. When we’re talking about a job I’m interviewing your company every step of the way. One key question I’m always asking myself is “does this seem like a hiring process that would hire people I want to work with?”. I don’t get a huge chance to interview my co-workers when I’m coming into a new place so my only real guide as to what type of folks they are is… the hiring process.

If your hiring process starts by selecting people based on how they rate skills, think of who you are excluding. Also, think of who ends up at the top of that list - arrogant, self-important folks who think they are experts in everything. Perhaps the HR person is clever and is looking for folks who rate themselves poorly to suggest humility but there are better ways.

I want to work on a team where you hire people because they fit well into the team, because they have passion about their job, and because they can learn. I don’t want you to hire someone who already thinks they know everything because every conversation with that person is going to be an argument about why something new isn’t as good as what they’ve already done before. People do what has worked for them in the past unless they are tinkerers who like to try new things - in which case they probably don’t become experts, they become generalists. The best folks I’ve worked with didn’t know how to do most of their day to day tasks when they started the job. They learned, they asked questions, they were humble and eager and engaging. Those are the people I want to work with.

Yes, I’m restricting my potential job opportunities by doing this - which is exactly the point.

I'm letting my CISSP lapse

2012-06-18T21:59:27-05:00

I took a 5 day course, studied for 2 months, and spent quite a bit of time stressing about passing the test for my CISSP. It was a good experience and I’m glad I did it - but the time has come to let it go. The problem is, it hasn’t done anything for me and it requires a fair amount of ongoing effort to maintain. Doing the work to keep up on the CPE’s is fine if you are in the security industry - I’m not. I was at one time, but not anymore and it’s not where I’m interested in being. I can fire up SANS webcasts and let them run on mute in the background on my computer to get CPE’s - but that’s just lame.

So, I’m letting it go. I’ve thought a lot about this and it would be different if I had ever had an opportunity that appeared to be gained by having the certification. I haven’t. If the knowledge I gained from the certification helped me do my job. It doesn’t. If people even cared that I have the certification. They don’t (except for one guy in Korea who I had BBQ with at 4am, he seemed excited about it). And lastly, if I ever thought I was going to go after a security job - but I’m pretty sure I’m not.

I think certifications have their place and I applaud anyone who goes out and earns them. I just personally have better things to do with my time than maintain something that really hasn’t added any value to my career. I encourage others to seek out certifications that do add value to their career and I encourage those who help maintain certifications to keep them evolving & continue to challenge folks to get them and keep them.

Embedding Ops members in Dev teams - my recent experience

2012-05-27T06:00:53-05:00

For about 2 months I was sitting with a dev team while we worked through how to build a new service which will be continuously deployed. I wanted to share my experiences here because I’ve read both positive and negative opinions about doing this and I’m not sure there’s a single right answer. It was certainly an interesting experiment and one I may repeat in the future, with some modifications.

This started when I began attending this teams daily stand up. My goal was to get more involved with a single team in dev to get a better idea of how the process worked in general. This team was one of our two Infrastructure teams which focus on scalability, stability & performance enhancing changes. Initially this team was creating a new Web Services API service which I wrote a little bit about here.

Eventually that service was set aside and the team moved on to a new Authorization and Authentication service. For this new service the decision was made to use Continuous Deployment. We were already doing fully automated deploys at least once per week but there was a bit of a jump to giving the developers the tools they needed to deploy every commit including monitoring & deployment automation changes.

I had also noticed, leading up to this, that the few times I had sat over with the team I was immediately more involved in discussions - they asked me questions (because I was there) and I had the option of attending planning sessions. There was literally a 20 foot difference between my own desk & the desk at sat at “with them” but it made a world of difference. As such, I talked to my management about sitting with that team all the time and they agreed to try it.

Now, this team is a bit unique. The team is constructed of a handful of developers working on the code but it also is the home of the Build & Release guy as well as our Sysadmin who manages the testing infrastructure. Sitting with this team gave me an opportunity to not only be involved in the development of this new service but to also become more involved in the Build & Release process, getting familiar with the day to day problems that are dealt with as well as pairing with folks to work on our puppet configurations which are shared between dev & prod. This team structure, along with me, also made them uniquely suited to tackle the Continuous Deployment problem (at least for this service) completely within a single team.

As part of the Continuous Deployment implementation we wanted to make it as easy as possible for developers to get access to the metrics they needed. We already had splunk for log access but our monitoring system required Ops involvement to manage new metrics. So as part of this new service we also had to perform a spike on a new metric collection/trending systems - we looked at Ganglia & Graphite. We weren’t trying to tackle alerting - we just made it a requirement that any system we select be able to expose metrics to Nagios. I worked with the developers to test out a variety of ways for our application to push metrics into each of these systems while also evaluating each system for good Operational fit (ease of management, performance, scalability, etc).

Throughout this process there were also a lot of questions about how to perform deployments. How many previous builds do we keep? When and how do we rollback? What is our criteria for calling a deployment successful? How do we make sure it fails in test before it fails in production? What do we have to build into the service to allow rolling deploys to not interrupt service? The list goes on - these are all things that you should think about with any service but when the Developers are building the deployment tools they become very aware of all of this - it was awesome.

After about 45 days we had the monitoring system selected & running in production and test, we had deployments going to our testing systems and we were just starting to deploy into production. We now had to start our dark launch, sending traffic from our production system to the new service without impacting production traffic so we can see how this backend service performs, whether it is responding correctly to production traffic & generally get a better understanding of behavior with prod traffic. Today this service is still operating dark as we tweak and tune a variety of things to make sure it’s ready for production - again, it’s awesome.

60 days in things started winding down. We had been dark launched for a few weeks and largely the developers had access to everything they needed - they could look at graphs, logs, if they needed new metrics they just added them to the code and they showed up in monitoring as soon as they deployed. We got deploy lines added onto the graphs so we could correlate deployments with trends on the graph - more awesome. However my work was winding down, there were fewer and fewer Operational questions coming up and I was starting to move back toward working on other Ops projects.

As I looked back on the last 60 days working with this team I realized the same 20 feet that kept me from being involved with the development team had now kept me from being involved with the Ops team. I was really conflicted but it felt like the healthy thing to do would be to move back over into Ops now that the work was winding down. I immediately realized the impact it had as people made comments “wow, you’re back!”… seriously folks, I was 20 feet away! You shot me with nerf darts!

So now I’ve been back over in Ops for a few weeks and there has actually been a change - I’m still much more involved with that Dev team than I was from the start. They still include me in planning & they come to me when there are Operational questions or issues that come up around the service. However, that 20 feet is there again and I can’t hear all the conversations and I know there are questions that would get asked if someone didn’t have to stand up and walk over. Our Dev teams tend to do a lot of pairing and as a result aren’t often on IM and email responses are usually delayed - pairing certainly cuts down on the email checking.

Was I happy I did it? Absolutely. Would I do it again? I think I would - but I would constrain it and set expectations. I think the physical proximity to the team helped a lot to move quickly and toss ideas around while the service was being developed and decisions were being made but it did have an impact on my relationship with the Ops team that I wish I could have avoided. I think continuing to move back and forth - spending time with the Ops team would be helpful. I actually did spend my on-call weeks (every 4th week) in Ops instead of sitting with the Dev team, but I would try to find some time during the 3 weeks in-between to be over there too, it was just too much absence.

All that said, I think overall the company and the service is better for the way this turned out and for me personally it was a super insightful experience that I wish every Ops person could try sometimes.

Monitoring is pretty cool - AssimMon

2012-05-26T12:18:03-05:00

Yeah, I know Monitoring Sucks, but it can be pretty neat as well. It’s a hard problem, so there are all kinds of interesting approaches to it. A new one I came across recently was the Assimilation Monitoring Project. I met Alan Robertson (Linux-HA founder) at a recent Cloud Computing meetup and in the post-meetup discussions he talked a bit about his project. One of the things he mentioned was that our monitoring systems spend the majority of their time (proportional to the quality of your system I guess) detecting and reporting that everything is ok. His project aims to distribute that task across your systems in a way that scales to quite large infrastructures & is inherently redundant.

I’m not sure how ready it is for prime time, but take a look - it sounds pretty interesting.

I think I’ll always have an interest in monitoring problems - not because I have to as part of my job, but because the problem has such a wide variety of potential solutions with different benefits. It’s also one of those areas that’s very rarely just plug and go, you have to architect it like any other service which keeps it interesting.

Why do we insist on consensus on the role of Ops?

2012-03-19T19:44:26-05:00

I’ve seen so many threads over the last few weeks about who should do what, why, and what you should do about it if you don’t conform. I don’t get it. Ops is a team in a company - there are lots of types of companies. Companies typically have a few goals:

1) Make money

2) Change the world, as long as we can do #1.

Lots of companies accomplish these goals doing things wrong. If you want proof, read Good to Great, there are oodles of examples of companies who didn’t qualify as “great” but who you would recognize as successful.

When wagon trains migrated families west across the US, the idea of driving 40mph, of crossing a state in a day, would have been crazy talk. Then came the locomotive.

When locomotives moved people across the country, the idea of a car making an interstate trip would have been crazy. It would be madness if everyone operated their own car. Then came cars, and road, and traffic signals, and road signs. This took time, lots of mistakes, lots of retrospectives, and year over year progress.

Progress isn’t made by conforming to the conventions of today, it’s made by pushing for something better. That’s what some folks are doing in Ops today - they are trying to push the limits and do what works for them. Others are observing these patterns and following suit. Still others are sitting back and saying “That ain’t right, my process works just fine”. Perfect.

It wasn’t necessary for automobile manufacturers to convince railroad operators that the car was the future. The car became the future because people adopted it, because it worked, and because over time the infrastructure that supported it became more mature.

As our tools get better, as our patterns become more and more repeatable, as we start to understand what roads & traffic signals & road signs we need for Ops to get out of the way of Developers making changes in production, things will move. In the mean time - talk about what works for you, why it works for you, and don’t bother convincing other people why it should work for them.

Sometimes it takes 2 days to do 2 hours of work

2012-02-21T23:37:21-06:00

I hear all the time:

“I could get that done in a few hours, easy”

“I could whip that up in 2 seconds”

So what? Instead of bragging about how awesome people should think you are because of how fast you say you can get things done, how about you ask some questions?

“When do you need it done by?”

“Do we have time to improve this other widget to make fixing your thing easier?”

We are so focused on trying to crank out as much as we possibly can, we sometimes think it’s better to talk about how quickly we can get things done. Instead, under commit & over deliver. If you have 2 days to get something done and you only need 1/2 day - spend some time improving something that will make your life easier. Understand what the expectations are before you commit & see if you can get in some extra benefit. If you hit a snag & end up taking 2 days to finish then nobody is disappointed, but if you don’t then you get some extra work done & still meet expectations.

I know folks prefer accurate estimates & like to fill your day with the stuff they want done - but don’t complain that you can’t get your stuff done if you aren’t under committing once in a while.

Getting out of the way - Monitoring

2012-02-20T06:00:25-06:00

In recent years I’ve come to view Operations as a traditional bottleneck that developers have become comfortable with. I think this fact is changing rapidly to give Developers more visibility into how their application behaves in production & to allow faster delivery of value into production environments through things like continuous deployment.

One of the areas Operation is often a bottleneck is monitoring. The traditional model is to have Ops ask Dev what metrics they need monitored & to set those up. This often means that monitoring can’t start until the metrics are available in the code, and then it isn’t for days or weeks after that when some Ops person has time to setup the monitoring system to pull those. This is broken and unnecessary.

If you are operating in a service delivery model where you have control over all the systems you monitor, you should be working to get out of the way. You should be working to make the monitoring happen automatically without Ops involvement. This doesn’t mean that Dev does all the work, what it means is that Ops selects monitoring systems that allow for discovery of new metrics & automatic collection of those metrics without additional incremental work each time.

Some of this is technology selection, some of this is architecture, and some of this is just doing the work. This does take work - but I would be hard pressed to find an example where the work required to set this up is not offset by the work saved in the long run not having to respond to every new metric that gets added. Below are some concrete examples of what I’m talking about - if you aren’t familiar with this read these.

Metric Collection

The yammer metrics library has made it really easy to expose your application metrics automatically. They have additionally provided hooks into tools like Ganglia and Graphite for pushing metrics to the monitoring system. As you look at how to scale a monitoring system, these are great tools to allow for that. Another popular data collection tool is statsd. The main idea is that you want to use collection tools that don’t have to have metrics pre-defined for them. If you give them a value for a metric they track it, that is all. The more often you give it to them, the more numbers they store.

Graph presentation

Ganglia is great for allowing you to programmatically define graphs and manage those via your CM system of choice, like puppet or chef. Another approach is something like Graphite which provides a rich and generic UI for taking whatever metrics you collect & combining them into a graph. Building custom dashboards and such in Graphite is where it’s strength is at.

Alerting

Nagios. We all dislike it, but it works pretty well. The main advantage Nagios has over more “intelligent” systems is that it can be configured through your CM system of choice. Additionally, Nagios has a massive community behind it. When building out Nagios or whatever you use, do your best to drive your configuration through CM and try to get things to the point where you don’t have to do any incremental monitoring work for each new system you add. New systems that are the same type as a system that’s already defined should just get monitored for “free”.

Summary

Think of monitoring like a service, like any other application in your architecture. You want it to discover what’s out there and configure itself as much as possible. Doing this isn’t completely simple yet - but it’s possible and if you set your mind to it you might even find a way to do it better that you can contribute back to the community. In doing this, try to get out of the way of your Developers and strive to have metrics they expose in their application automatically show up in your monitoring system of choice. Try to make it very low cost for them to add new metrics to see new information & you will probably be surprised at the amount of monitoring you get when all a Developer has to do is write the code to track the metric & it shows up in prod.

Overcoming Obstacles

2012-02-17T06:00:25-06:00

Three days a week, unless there’s something seriously preventing it, I scarf down my lunch and head down to the bouldering gym for an hour of climbing. This isn’t a post about rock climbing, but in the short time I’ve been climbing (about 8 months) I’ve found the whole practice to change the way I think about some problems. Rock climbing is a physical sport, but it’s also a very mental sport. Climbing difficult routes is a combination of strength, technique and determination. The process of learning is interesting as well, because you don’t hear a lot of people giving advice in the climbing gym. Folks lead by example.

So, I wanted to draw some parallels to my observations about climbing and working through obstacles in tech. We deal with new challenges every day and some can be pretty intimidating. We also learn and teach a lot, and I think there are some lessons about that as well.

Don’t allow obstacles to defeat you before you start.

When you approach a new route it can be intimidating, you aren’t really sure what to expect, what part of a route is going to be most difficult. This is true of approaching many problems, but just because a problem looks intimidating does not mean it cannot be overcome. There is a lot we don’t understand until we are working our way through a problem and telling yourself you can’t do this isn’t going to help. Make one move at a time & do your best - rarely is the situation so critical that you cannot afford to adjust as you learn.

When you miss, inspect and adapt.

The bouldering gym is full of big pads. Those aren’t there because nobody ever falls. Everybody falls. This is part of the process of challenging yourself, part of the process of trying new things. You go to the gym to fall because it’s safe to challenge yourself there & learn how to improve.

Too often I hear folks who are afraid to fall, afraid they might choose the wrong path when working through a problem in their life, their career, or some technical issue. It just isn’t possible to know the right path 100% of the time so don’t bother, do your best. When the inevitable fall happens, take another look at your moves, try to understand what went wrong, and try again a different way.

Inspecting and adapting to what you learn is one of the greatest skills you can learn. Freeing yourself to make mistakes removes a lot of barriers that you thought were there when they actually weren’t.

Watch others

In the Gym this means literally watching other climbers. Some climb with such grace that it makes things look so easy. This is true of a lot of things - so look at what others are doing. We all experience problems in different ways and we all solve them in different ways, learning from each other is key to progress. But keep in mind too that what works for one may not work for another. A tall person will climb a route much differently than a short person, they have longer reach, they also have a different center of gravity. Use ideas you see, but don’t get too upset if those same ideas don’t work for you.

Be patient

When you first start to climb, as when you first get into most things, there is a period of fast improvement. You feel great, you are learning fast, you must be awesome. As you learn more and as you start to approach more difficult challenges, your ascent will seem to slow. You are getting better, you are learning stuff, but it’s not as easy as it used to be. Once you’ve been doing this for 15 years, the problems that are hard to overcome aren’t about learning how to use some new programming language or learning to deal with some new technology, they are the finer points that actually make you better day to day. Those things take time to overcome, they are hard problems that require discipline and persistence like you have never needed before.

Climbers who have been climbing for many years will tell you that it becomes very hard to progress to the next level. Each progressive level requires significant improvement & a lot of work. You have to be patient & keep at it, you have to love climbing to climb, and you will improve.

Be Helpful

The only reason I am where I am at is because there were people who were willing to help me along the way. When I first stepped foot in the climbing gym there were people who showed me the basics. When I was clearly struggling with a route, there were people who climbed it & offered advice. When I have had problems finding that missing semicolon in a sea of code, there have been others willing to lend a 2nd set of eyes to find it.

We need each other to overcome obstacles and we each bring a different set of skills to the table. Your being helpful contributes to that, just as you have leveraged others helpfulness to get where you are at. Give back and help out.

Be Nice

It’s easy to be arrogant. It’s easy to tell someone that your challenges are more difficult than theirs. It also serves no one but yourself. Be kind as you work your way through your challenges because relationships matter more than any ability you could ever learn.

Keep decisions low cost so they are easier to make

2012-02-16T06:00:27-06:00

Which question is easier to answer?

1: Select the storage platform which will serve all our needs for the next 3 years up to 10PB.

2: Select a standard disk drive to use in the next 5 servers we purchase.

Most people could answer #2 pretty easily with a little bit of information. It might take a day or two to figure this out. And if you get it wrong, what’s the worst thing that happens? You’re replacing 10 disks or so. #1 though, that’s harder - that requires knowing what you are going to be doing for the next 3 years, how much money you can spend, how much flexibility you are going to need, etc.

The cost of being wrong for question #1 could be millions of dollars.

The cost of being wrong for question #2 probably means a few thousand dollars.

You don’t always have the luxury of changing the question, but when faced with a decision that seems to be hard to make, look for ways to make it easier. One of the ways to make a question easier to answer is to lower the cost of being wrong. For question #1, how can you answer that question for the next year without excluding the possibility of expanding to 3 years? How can you get more data about your needs by starting small, gathering data, building understanding? How can you constrain the scope, not solve all your problems but solve the most pressing ones?

Try not to solve all your problems at once because there’s a good chance you aren’t going to succeed. Pick a problem, and do the simplest thing that works to fix that problem.

Work-life balance is personal

2012-02-15T06:00:07-06:00

I was once (or three times) told that I was working when I should not. Usually this happens in response to a 10pm email, or an email on the weekend, or three of them. The thing is, that’s an important part of my balance and if I don’t do it I get thrown off.

I like the idea that companies try to embrace work-life balance, the ideal that says you should have a life outside of work. What I do not like, however, is that anyone thinks they know what time of day I should be working. Just because someone 50 years ago said that 40 hours a week is ‘normal’ doesn’t mean it’s appropriate for everyone. I get ideas at 10pm & I want to share them while I’m inspired.

Inspiration & opportunity come my way when they do, and when they come I have to decide if I want to make my move and get something done or wait until I’m “on the clock”. Doing the latter almost always means not doing it at all. Inspiration, after all, expires rather quickly.

What if work-life balance meant being honest with yourself and your employer? What if it meant that when there was an opportunity, you jumped on it, and for doing that you were given the flexibility to take that time back when there were personal opportunities outside of work? A number of companies do this already, it’s just informal and largely based on trust. The problem with this is that when it becomes a problem because of an individual, it has to get corrected for the whole team. Usually that correction comes in the form of “core hours” or some other lame blanket of consistency placed on the many because of problems with the few.

Part of the reason individuals see a problem in my work habits is because they feel pressure to conform to their interpretation of my actions. If I send a note at 10pm, that doesn’t mean that I am working 8 hours during the day + 2 hours at night. Some days it might mean I worked 10 hours, other days I might put in 6. The problem is that people feel pressure to do the same. They see someone else working “overtime” and feel like they need to. But they don’t know what I’ve been doing - they just see me working off-hours and assume it’s “overtime”. This turns into the ultra-competitive work-all-day-and-night environment that some companies have. That’s not what I’m talking about. That’s not what I do.

I have the good fortune of having work that overlaps with my passion, which is often my hobby. My hobbies contribute to my work, and vice versa. This blurs the distinction between work & life. My wife struggles with this - “I can’t tell if you are working or not when you are on the computer”. To which I ask “Why does it matter?”. Whether I’m working for my own goals, or working to get paid, the parameters are really the same. She doesn’t like that answer because she doesn’t want to interrupt me when I’m working, but if I’m not working it’s ok. The truth is, it’s always ok - and I’ll tell her (nicely, I hope) if it isn’t.

Maybe if we were more honest about how we worked & more open about what we expect from each other it would be easier to work this way. If we managed more based on achievement than hours or individual tasks. Maybe then we could all work better.

Are you making blameless, data driven decisions?

2012-02-14T06:00:58-06:00

On this day of emotion & emotional decisions, this post is about not making decisions using emotions.

When I hear people use words like “could be” or “should be” I start to wonder if there’s enough data on the table to make a good decision. It’s true that sometimes you just don’t have the answer, so you have to ask more questions. How can we learn more to make this decision easier to make? How can we make this decision easier to change if we get it wrong?

Decisions made in the absence of data have another problem, the only collateral is personal responsibility. You are trusting someone’s gut, or someone’s experience, or someone’s opinion. When things don’t go right, you blame that someone.

If you are making a decision that is hard to change, it should be made when the and answer to questions are “it is…” or “it is not…”. You use those words because you have confidence you have tested & have data to back it up. You should be saying “We know” and not saying “We think”. Saying you know without data is just being dishonest. I don’t care if you’ve done this before, the people in the room have only your credibility to trust. Solutions don’t work in different environments for a variety of reasons, but if your credibility was the basis for a decision it will be the only thing that gets blamed.

Try to make decisions easy to change.

Try to make decisions based on fact, not opinion.

Try to make decisions that do not rely on your credibility.

If you aren't using feature toggles, start... now

2012-02-13T06:00:39-06:00

Search for ‘feature toggle’ in Google, check out the results. The simple fact is that branching using a revision control system still has its place, but its place is not controlling when you release a feature to your customers. Feature toggles create a distinction between deploying your feature & making that feature available for use. They also remove the requirement that to disable a feature, or to go back to ‘old behavior’ you have to rollback your deployment to an older version of code. There are lots of other benefits too, as well as some challenges.

Bottom line though, if you aren’t using these you need to really seriously consider whether they would be a benefit. If you control your software release & you operate a multi-tenant system, and you want to increase the amount of control you have around the features you release, you need to be using these.

Here are some related blog posts:

http://code.flickr.com/blog/2009/12/02/flipping-out/

http://codeascraft.etsy.com/2010/05/20/quantum-of-deployment/

http://www.rallydev.com/engblog/2011/12/09/the-awesomeness-of-feature-toggles/

http://www.rallydev.com/engblog/2011/12/12/the-best-part-of-feature-toggles-removing-them/

http://www.rallydev.com/engblog/2011/12/14/testing-feature-toggled-code/

http://www.rallydev.com/engblog/2011/12/20/feature-toggles-branching-in-code/

Establishing ownership in Ops Teams

2012-02-05T10:28:54-06:00

I’ve been having some discussions about this lately so figured I would write something about the topic. Being a member of an Ops team can be pretty challenging at times. The job can be high pressure and often it feels like you spend all your time fighting fires, shaving yaks, etc. One of the difficult parts of being in Ops is that it’s often hard to put your mark on things, to use your skills to leave a lasting impression.

The reason it’s hard to leave a mark isn’t because there’s a lack of work, but because the work changes so frequently that influencing the long term outcome of a project can be hard. This can often be even more difficult in Operations teams following Agile methodologies because the work is broken into smaller stories and those stories may get worked by multiple folks. Even within these teams though, there are individuals with skills in certain areas and often there is more than one person with passion for a particular topic. Someone who’s passionate about a topic is more likely to do a great job, in my experience, and so we should see how we can leverage it.

Roles and Responsibilities Matrix

One successful tool I was shown was a Roles and Responsibilities matrix. The goal of this is to establish some basic ownership of components within an infrastructure so that individuals can focus their work. This often happens naturally in teams, but doing this formally accomplishes a few important goals:

Allows individuals with no experience, but with an interest, to raise their hand and work with new things.
Allows the team to agree on who is responsible for what infrastructure pieces. This is not sole ownership, but more about establishing expertise & creating less contention over decisions.
Helps you, as the manager, formalize who to work with on specific issues.

The matrix is pretty simple, for each component (you can partition this however you want) you define two roles, a “P1” and a “P2”. These are the primary and secondary points of contact for that component. But there’s more to this than just having a primary and secondary:

P1: This person is the current “non-expert”, the trainee. All escalations for this component should go to them first. If they don’t know the answer it’s their responsibility to work with the P2 & resolve the issue. In this process, they learn.
P2: This person is the current expert, the trainer. They understand that they are P2 and are to work with the P1 on issues where they need help.

I have also observed this setup where there’s only a P1 and they are the expert because there just aren’t enough folks to have a P1/P2 for that component (or it’s not a priority). Another reason for the P1 to be the expert is if the system is going through a lot of changes and you want someone to keep tight reigns on what changes are made.

Here is what an example matrix might look like

Looking over this, each person is a P1 for one component & a P2 for some other. In a perfect world it works out like this, but the world aint perfect. Do your best with what you have - but try to setup something like this.

This is usually established during a meeting every quarter or every 6 months. You walk through the list of functional areas and ask for volunteers. This more often than not ends with very little contest, but in the event where there are concerns about who is P1 or P2 you should try to understand why it’s important to each person to have a role in this, what they want to accomplish, and consider what other areas they also want to accomplish things in. Often, after discussing their vision on this component along with other stuff they are working on it’ll become more self evident who is the best P1 & you can get agreement.

Defining cross-functional areas

The matrix above works well, but the first question from folks is usually something like “what about monitoring, if I own that does it mean I have to do all that work for everyone else?”. The answer is “no” in most cases. There are some functional areas which are pretty clear & mostly self contained but there are others which cut across all the other areas. Examples where something intersects with everything else are Monitoring, Networking, Configuration Management and sometimes things like Storage, depending on your architecture.

For areas where your area of expertise is a dependency for others there needs to be shared ownership of those tasks. I generally look at it this way, using Monitoring as an example:

The P1 is responsible for overall architecture & infrastructure, training, documentation & escalations for that system. They are responsible for enabling the other team members to use the system effectively & for bringing any major changes to the team for review & consensus.
The P1 owners of other components are responsible for integrating their systems with monitoring, for writing any monitors, and for establishing meaningful metrics & thresholds for that system.
Both P1 owners work together to make sure any monitoring / metrics are done in a consistent way that is inline with what the team has agreed is the architecture.

In this way you are avoiding making the monitoring owners job suck by having to spend all day writing monitors for a million different components, but they have ownership of the overall success of the monitoring infrastructure. Individuals who own other components are making decisions about how best to monitor their own systems within the constraints of the best practices for the monitoring system & they can work with the monitoring owner if they want to break new ground on doing things a different way.

Working outside of Operations

One of the most important roles Operations plays (in my opinion) is in working with Development as closely as possible. This is becoming more and more obvious and more teams are starting to give it names, like DevOps. Some Ops folks are better at this than others and some will go out and find Developers to work with and others need to be prodded a bit.

Defining clear roles for individuals in Ops is a good way to force this collaboration. By assigning one Ops person to an upcoming Dev project & setting clear expectations around that role, you help foster their involvement and empower them to start working with other teams. That Dev team becomes a functional area, and they get a P1 & P2 like any other component.

What I would typically advocate for smaller Dev organizations is integrating one Ops person per Dev team if you can. This means that Ops person attends stand-ups, they go to planning meetings, and they are familiar with all the stuff that Dev team is working on. Should there become a need for Ops related work (or communication, which is always needed), the assigned Ops team member is responsible for that role. They aren’t necessarily responsible for all of the work but they are responsible for making sure the work is communicated & making sure it gets done.

Another approach is to assign Ops team members to individual projects. As projects arise, team members start to attend those meetings & start to get involved with any stand-ups and work around that project. I don’t like this approach as much because it relies on the Dev teams reaching out and saying “Ok, we’re ready for an Ops person now” most of the time - and that often happens late. Having Ops members already in position inside teams gives you much earlier warning and helps shape the end result much earlier.

Tracking & Communicating work

Now that everyone is working on their own projects, there will be a tendency to communicate that work less often & less completely. It takes some work to avoid this but it’s actually not all that hard. The important aspect of this is that each team member is talking about what they worked on each day at stand up & are being clear about their priorities during planning sessions. How you achieve this is up to you - but I’ll throw out some ideas.

Kanban works well as a visualization tool for work in progress. From an Ops perspective, I think that’s where the role of Kanban starts and ends. Operations is an inherently interrupt driven team and while many organizations get out of that mode through lots of practice - if you are at that point you probably don’t need my help in tracking & communicating work. Where I have seen Kanban work really well is in prioritizing work during planning (abc must come before xyz, move the card) and in visually showing what you did, what you are doing, and what you will be doing next.

Daily stand-ups are really, really helpful. Things change day to day in Ops teams and taking 10 minutes each morning to get everyone in sync with what’s going on is a huge help. Identifying blocks and talking about how to clear those is a big part of this. When everyone is there talking about things, saying “I’m blocked waiting for xyz” is an opportunity to get that problem solved today.

Also documenting proposals using a shared document system like Google Docs is a massive improvement. I can write up a proposal for something and instead of asking for feedback, people can add it right to the document - they can make comments, etc. We get together for a 30-60 minute meeting to review the document & the feedback and we take a shot at a final proposal. If there are still open questions we go back and answer those. The key is that much of the work is done asynchronously rather than asking that everyone bring their best, most un-distracted thoughts, into a meeting.

Rotating roles

Lastly, with all of this, there is change. Nobody wants to be stuck in the same role for years - people in Operations want to learn new things, they want an opportunity to take something that needs improvement and leave their mark on it. In every infrastructure there are some cool projects and there are some lame projects. There are also those parts of the system that are just a pain in the ass to maintain & nobody wants to do it. It’s important to rotate these around.

What has worked in my experience is a periodic review of the priorities. You start with a review of work in progress so that folks know what they are signing up for if they want to tackle an area they aren’t working in today. Then you wipe the slate clean & go functional area by functional area asking who wants to be involved.

The trick with this process is to try to allow folks who have projects in flight to maintain that responsibility while giving someone else a shot at learning about the system. This is where the P1/P2 roles can really be leveraged. If you are re-building your network and you really need the same guy to maintain his momentum in that project - he becomes the P2, continuing that work. You assign a new P1 (if someone new wants to be involved) and you have them tackle the day to day interrupts. The two members work together on it and the new gets to learn while the old gets to finish their project.

If a functional area has no work in progress and you really want to move something new forward there, find the person who’s passionate about making that change and make them a P1. Find a P2 that can help enable them and let them go for it.

Wrapping up

Ownership is an important part of any job and in Operations it has been the light that keeps me coming back. Giving that ability to every member of your team is important, and hopefully this gives you some ideas about how to do that.

Thanks Seth

2012-02-03T23:32:06-06:00

There are few people I can point to on the Internet and say “you made an impact”. A few have, and countless others have contributed in various ways to the ideas I have today. No other however, has had the impact of Seth Godin. I’m sure you know who he is already, if not then I hope you gain something from this.

I try to read every post - I don’t always manage, and sometimes I read them out of order. Today I read this one. Like so many others, each having its own impact on me, this one made me pause and think “is this what I’m doing?”. Seth puts something out there every day. He always shows up & I always look forward to it.

So I just wanted to say thanks. Thanks for a variety of great books. Thanks for introductions to great people. And thanks for being a daily source of new thoughts.

That is all.

Despite your best efforts, some people are just...

2011-12-18T23:30:44-06:00

… asses.

You should try to be aware of your own faults, try to improve yourself and try to be patient with others. You should try to find ways to be more clear & describe your position so that others can understand it. You should work hard to overcome geographical barriers, differences in culture & background, we’re all different in special ways. You should try to put yourself in other’s positions every chance you get.

But despite your best efforts, some folks are just asses. That will never change. There are folks you just aren’t able to get along with.

Recognize this & don’t bother trying to change them - just try to avoid showing the worst in yourself in the process. Don’t be an ass too.

Find a way, some way, to still be awesome in the eyes of those who matter.

Asking for permission

2011-12-14T07:32:22-06:00

The decision of whether or not to ask permission comes up a lot in Ops. I’m sure you’ve heard that sometimes it’s better to ask for forgiveness than to ask for permission. As with most things in life, try to put yourself in the other folk’s shoes and ask yourself how pissed you would be if nobody asked your permission. If you are a Honey Badger, skip this step.

Asking permission to make a change, as an example

When it comes to major changes I always communicate the change but I don’t always directly ask for permission - it depends a lot on how much I feel like getting feedback. Sometimes I know what I need to change is the right thing to do and I just want to know if there are gotchas, other times I want to solicit feedback and know if others think there’s a better way.

Two ways of asking that question:

“Hey all, I am planning to shutdown systems a,b & c today. They’ve been removed from service for a week. Let me know if there are any problems this will cause. I’ll be shutting these down at 2pm MDT”

vs. this

“Hey all, I would like to shutdown systems a,b & c - is this ok? I want to make sure everyone agrees with this before I move forward. I’ll bring it up during stand-up as well”

What’s the difference?

The first is saying “I’m doing this, let me know if I’m going to stumble in the process”. It’s not so much asking permission as asking for advice. The second is saying “I’m not going to do this until we all agree”. I generally prefer the first approach when I can get away with it - but often I will use that after I’ve asked about the change using the 2nd approach. It’s important to make others feel like they have an opportunity to say “hang on, I have a question” and provide input - but in some teams it’s also important to not allow habitual questioners to derail something that needs to get done.

On the other hand, don’t just start using the first approach because you always get questions. If you are getting legitimate questions it may be because what you are doing isn’t being communicated well enough or there isn’t consensus on the team about what you are doing. Respect other folks opinions & make sure they have an opportunity to be heard.

The exception, facilitation

When you are diving a meeting the power is a bit different and so is your need to ask permission. When you are driving a meeting, and in particular if you are asking for a team to participate, you need to ask permission. I run a lot of post-mortem type meetings in my current work and in that capacity I’m constantly asking these sorts of questions:

Does everyone agree that this is an action we should take?
Do we agree with the timeline in this post-mortem, are we missing anything?
Is everyone comfortable that we’ve captured all the actions from this event?

And the list goes on… When you are driving any meeting you have a lot of power over the outcome of the meeting, and so you need to weight decisions more toward the team and less toward yourself. As soon as you say “I think we should do…” it becomes much harder for someone else in the room to stand up and say “I disagree”. Despite that, some people still do.

The other exception, ownership

I don’t usually ask “Is everyone ok if I own this?”. If I want to take responsibility for something and own it’s completion I say something like “If nobody else is already working on this I’ll take it, let me know if you are interested in helping”. Ownership doesn’t have to mean you are going to do all the work, but that you are taking responsibility for it getting done. I haven’t found many cases where folks are unwilling to allow someone to own getting something done, but I have found plenty of cases where people want to have input into how it gets done.

Wrap up

So, go forth, ask permission where appropriate, and ship it.

Operation Bootstrap

DevopsDays 2013 - we are avoiding culture, why?

Good & Bad patterns in Development and Operations

Culture & Communication

Good:

Bad:

Eliminate Manual Processes

Good:

Bad:

If it moves, measure it

Good:

Bad:

#1 Culture & Communication

Main points

#2 Obsessively eliminate manual process - let computers do what they are good at.

#3 If it moves, measure it

Continuous Deployment: Are you afraid it might work?

I think you should listen to me, but that’s irrelevant

Dude, I would never do that…

Reality check - you are deploying more often than you think

Wrapping up - why so much focus on Continuous Deployment?

Further Reading

DevOps is Culture - so is a lot of other stuff...

Rate my skills: no

I'm letting my CISSP lapse

Embedding Ops members in Dev teams - my recent experience

Monitoring is pretty cool - AssimMon

Why do we insist on consensus on the role of Ops?

Sometimes it takes 2 days to do 2 hours of work

Getting out of the way - Monitoring

Overcoming Obstacles

Keep decisions low cost so they are easier to make

Work-life balance is personal

Are you making blameless, data driven decisions?

If you aren't using feature toggles, start... now

Establishing ownership in Ops Teams

Thanks Seth

Despite your best efforts, some people are just...

Asking for permission