VMOS

Imagine a seven year old playing the piano. She hits every note like it’s the only one, taking long breaks between each note. The play drags and listening to the singular notes is a pain. Instead of music, all you hear is a bunch of individual sounds, each one rivaling with the others to be the loudest one.
Now imagine the same play performed by the same kid five years later. The notes flow like a river, the emphasis not on individual sounds but on the whole sequence at once. Listening to the piece is pure joy, because every note works together with the others to create a beautiful experience.
Distributing your software development through separate departments is like a seven year old playing the piano. Every department works on its own, rivaling with the others to be the most important. The output is a pain for your customers and the quality is really poor. But how can we create an organization where the individual parts play nicely together? One way of making departments play together are feature teams. Let’s see how this could work out.

Feature teams are cross-functional teams

They consist of team members from every department involved in feature development: business analysts, developers, quality assurance, operations, and so on. Ideally, they’re co-located – working in a big room together or at least in the next room. This shortens communication paths and speeds up necessary information exchange.

Feature teams are responsible for a certain set of features or a specific service. At amazon.com every part of the website like recommendations or reviews are separate services developed by individual feature teams. Each team is fully responsible for the feature, from idea to operations. Any challenges that arise within their specific feature must be solved by the team.

Solving challenges is simpler in smaller teams

If you form feature teams you should keep them small. A famous quote from Amazon: “If a project team can eat more than two pizzas, it’s too large”. Keep your team between five to nine people. This will keep the complexity of inner-team communication as low as possible. You scale your projects by combining multiple small teams – each one fully responsible for certain features.

Embed every required skill within your teams to create a successful feature. This isn’t possible with all skill sets, but you should limit external party support as much as possible. External dependencies always slow down a team and increase complexity.

It’s all about reducing complexity

Trying to drive a feature through multiple departments can be quite a challenge. Every department has their own goals which usually conflict with other departments. Even if they want to help, they have a hard time understanding exactly how.

Building feature teams tears down those departmental barriers and joins people from every department into one team. This helps to create empathy and, more importantly, trust – the number one factor to speed things up in complex environments. Without trust, blame games and CYA tactics are rampant. Only if people know each other, have the same goals, and work together are they more likely to trust one another.

That trust thing sounds great, but won’t work for us

Let’s say you already have feature teams. This is often the case in so called “matrix-organizations” where every employee is part of a certain department but works in multiple projects. And with multiple projects, the problems start. The employees are supposed to serve multiple bosses: the project leads as well as their department head. Because they’re assigned to multiple projects, they usually find home in their department and give it the highest priority.

But giving the highest priority to the department is not what you want. Therefore it’s important to assign people to one and only one project. Make that project (and the feature team driving it) the home for your employees. The department should mainly serve as interest group.

Feature teams take responsibility

They’re fully responsible – end-to-end – for a certain feature or service. They’re small, co-located and they have all the needed skills to be successful. Feature teams act as the home base for employees as departments fade into the background providing mentorship for honing one’s skills. But the main, business contribution happens in the feature teams.

If you setup your organization like this, you’ll begin to see a nice flow of great feature releases – like a well played musical piece.

Tom Check

“But our customers don’t want 10 new versions a year. The last release alone had over 600 bugs!” retorts the hotline manager.
“How about a small update with just a handful of bugs?”

Your big-bang release is scary. It’s full of issues and weird, new features that nobody understands. It requires documentation and training and who the hell has time for all that?

Monthly, bite-size updates will have fewer features requiring less support (pro-tip: less code == less bugs). Speeding up your release cycle also allows quicker response to customers’ feedback. You’ll finally feel your company moving in the right direction again.

Of course, it’s easy to say. But how can you actually achieve this positive flow? Follow these key points and you’ll be well on your way.

Keep it Simple Stupid

• don’t spend three months gathering specs
• create immediate user value (MVP) and prepare to iterate based on customer feedback

Cross-functional Team Empowered to Release

• Product owner, QA and devs write acceptance test together up-front
• devs code until test is green
• sysadmin deploys live

Automatic Distribution Channel

• easy way to get updates out quickly and reliably
• painless for the customer to install

Switch on the Feedback Loop

• know about problems before the customers (and have fixes ready to deploy)
• listen to what customers are saying about the new features and adjust for the next update

Bonus: Business Shifts to Monthly Subscription Revenues

• align the business with the development schedule
• inflexible departments don’t work well with motivated project teams
• make sales with a carrot not a stick ($19.99/mon vs $3000/yr)

GrahamPics1

Out with the Lion and in with the Lamb

Which of these two beasts would you rather handle? Thinking about opening up a petting zoo and charging money? I’m pretty sure not too many folks would pay to pet an enraged lion, yet that’s exactly what enterprises expect.

It’s time to tame our releases. Make them something our users actually look forward to.
“Would you look at that? What an adorable little thing!”

Burning Image

Monday morning on the highway. Your speed: 0 mph. You’re stuck in the usual rush hour traffic jam because the capacity of this road is exceeded. And it’s now obvious you’ll reach your destination much later than if the road were empty.

But what happens if you exceed the capacity of your development team? What happens when you cram in more features than the team can develop? Your features will get stuck in development. Let’s see how this can happen.

What happens if you push too many features into your development pipeline

Let’s discuss three negative outcomes of overloading your pipeline:

1. You lose focus by increased task switching
2. Managing the waiting queue costs a lot of time
3. Big releases create big headaches

Having too much upcoming work leads to task switching

When an avalanche of requirements hits, you have the urge to do everything in parallel because everything is important. You dutifully start with the most urgent task, only to have the second most important thing blow up in your face. Switching focus to the new number one in your list lets you watch the next three things go off like an ill-timed firework display.

When you’re constantly switching from one task to the next, you’ll hardly finish any one of them. Every single task stays on your todo list longer than it should. And I’m not even talking about the time you lose adjusting to every new task. Just the fact that you’re hopping from task to task increases the lead time of each dramatically. Instead of getting your overloaded pipeline cleaned up, more and more stuff piles up in front of you.

The second problem is managing the waiting queue.

If more and more stuff piles up, you waste time managing that queue

Again and again, you revisit and re-prioritize your ever growing list of tasks. Every time you go through that list you need to remember the details of each item, it’s current status, and so on. This takes a lot of mental energy and a considerable amount of time. That time and energy are lost instead of used to finish things.

And while you’re not getting anything done, guess what? That todo list grows longer and longer resulting in even more queue management work. A vicious cycle which is only made worse with today’s tools. We’ve got a host of bug tracking tools which let you manage hundreds or even thousands of open issues. I was once very proud of having over 1400 open issues sitting in our bug tracker. But it was an ordeal to go through even the upper part of that list again and again. We finally scrapped it and never looked back.

And that brings us to the third ill effect of overloading your development pipeline: big releases.

Dealing with too many features in parallel means preparing big releases

And assembling big releases means a lot of complexity. This complexity increases management overhead and reduces quality because no single person has a complete overview anymore.

Instead of knowing exactly what went live (and what might have caused a certain bug), you’ve got a big batch of stuff going live all at once and no chance of ever finding out why something broke. This is a very bad situation.

But it gets even worse than that: with growing complexity, I’ve found the number of bugs increase more than just proportionally. That means if you’re releasing double the amount of features, you’ll have to deal with way more than double the problems. Remember that nice task list – your “development pipeline”? Feel free to cram these release issues right on top.

“Yes, quality issues are clogging my pipeline, but there’s nothing I can do about it”

I know, it looks like that. But I’ve seen teams move from big releases with weeks of stabilization phases to continuous deployments. By creating a pull system and focusing on flow it is possible to break the vicious cycle.

These things get you into a vicious cycle

If you overload your pipeline, things will get worse because of more task switching, increased management overhead to manage your todo lists, and the unfortunate dynamics of big releases.

Create a pull system and focus on flow to break that vicious cycle and you’ll never feel like being stuck on the highway during rush hour.

What are your experiences with having too much stuff to do? Please share it with us in the comments.

Wikipedia

A rope. Eight people on either side. “Pull!” And then it begins: both parties are pulling in their own direction. A tug-of-war has started.

Imagine your developers and sysadmins as those two parties starting that tug-of-war

Each group has different goals. And having different goals leads to each party pulling in another direction. How can this happen and what to do about it?

In most organizations every employee has a set of personal goals

They usually bubble down all the way from the CEO (who get’s his goals from the board) via the middle management down to each and every developer and sysadmin. In most organizations development and system operations are separate departments and have separate managers. Both might report to the same C-level executive but both will get a different set of goals.

Those goals seem to make sense from the point of view of the C-level executive

But things get out of control when goals are passed down the hierarchy. What began as a nicely orchestrated company roadmap, entangles into a distorted mess of misaligned, even conflicting goals for the teams responsible for execution.

The mess thickens faster than you’d imagine

Like when playing Telephone (aka Chinese whispers), the original intentions are changed after filtering down through a few levels in the hierarchy. Just one or two levels are enough to end up with conflicting goals.

Conflicting goals lead to a tug-of-war

To avoid such a tug-of-war you need to set coherent goals. The most important step towards coherent goals is the harmonization of goals for all team members involved in a business process. With everyone finally pulling in the same direction, you can create an aligned DevOps organization.

You can harmonize goals by discussing them openly across the teams

Letting teams play an important role in defining goals makes sure that everyone understands and embraces the strategic business goals defined by top management.

Understanding the strategic business objectives helps to create harmonized goals

Once everyone understands and embraces the strategic business objectives, you have a harmonized set of coherent and realistic team goals. You’ll avoid that tug-of-war and your departments will collaborate naturally. You’ll experience the dawn of a real DevOps culture.

Alasdair Middleton

There are DevOps tools and DevOps job ads. People talk about culture and sharing and being nice to each other. Sounds pretty fishy, right? The only thing missing is a DevOps certification and we’re done with the DevOps hype. Is DevOps really just a fad? Let’s take a closer look…

You’re frustrated with another department – again

They don’t seem to understand the simplest things. They just throw stuff at you which keeps exploding in your face. The only way to deal with the continual onslaught of their crap is to become cynical. You rant about those guys and try to blame them whenever you can. You are sick of working on their garbage again when you could be doing real work instead. The things that really matter to your department.

But this is why you get a paycheck, right?

Actually, it’s exactly where DevOps comes into play. Why should your day be filled with anger? DevOps tries to overcome the aforementioned obstacles. By encouraging collaboration earlier in the process, DevOps tries to smooth out the boundaries between departments. By demanding automation, DevOps tries to minimize chores and create more room for valuable work. And by encouraging sharing of problems and solutions openly, DevOps tries to dissolve the combative cultures of blame.

It’s up to you to decide whether this is a fad or not. What do you think? Please let us know in the comments!

AMagill

Investing into code improvement is a dual edged sword: on the one hand you know that if you don’t improve your code you’ll get slower over time. On the other hand improving your code does not deliver tangible value to your users. So how do you know whether you’re on track?

Track the time a user story needs from idea to production

If that time is getting longer you’re off track: your code is slowing you down. It’s time to invest more into improving your code.

Measure the number of bugs per code push

If that number increases, you’re investing too less into code quality. Build quality in by writing automated tests upfront and by streamlining your code base. Rip out over-complicated stuff which people fear to touch because it breaks every time they do.

Are you spending too much time gold plating your code?

If the above measures do not move in the right direction even though you’re trying to keep your code as clean as possible you’re optimizing the wrong stuff. Try to find those areas in your code which are worth improving and focus your improvement efforts there.

Don’t try to improve stuff which is never touched again

It’s only worth changing even the worst piece of code if you’re going to touch it again. While it sounds like a good idea to ensure that all your code adheres to your coding standards, that’s not feasible most of the time.

Make sure you put the right measures in place to find out whether you’re investing too much or too less into improving your code. What is your strategy to balance new features against code improvements? Please leave a comment below.

State Library of South Australia

London Stansted Airport, early afternoon – a huge crowd at the railway station. No trains – only people, and a lot of confusion and anger. Having just flown an hour from Munich, Germany, I was supposed to meet a client two hours after touch down. But here I stood, waiting with the crowds: the railway just broke down. After several hours of waiting, I was able to catch a bus and reached London city late in the evening – but the client meeting was long over. The whole trip (not to mention an entire day of my life) was wasted just because one leg in my journey failed. Do your users waste their time because you have failing legs in your product development journey?

Your customers suffer because of delays!

If you accept delays within your product development pipeline, your customers lose out on opportunities and waste a lot of time. No matter whether a feature comes late because you ignored it, wasted time gold-plating the code, or just let it sit in your code repository for weeks, your clients derive zero value.

Your customers only get immediate value from developed functionality when you avoid delays

You need to avoid delays throughout the entire product development cycle. Optimizing just one part (i.e. developers should do “agile”) is like flying a faster plane to London just to get stuck there earlier (and longer). If you want to ship features as fast as possible, look at every stage, not just your area of influence.

You can’t influence the way features are designed

But you can try to talk to your colleagues from product management and try to find ways to avoid delays. Can you do with less upfront specification? Would it be possible to start spiking complex technologies long before a feature is completely specified? Is there a way to break down features into smaller independent parts? Show them you’re ready to adapt the way you’re working to help the overall goal of shipping sooner.

Still playing the blame game?

If you are, take one step back and look at the overall process. How does blaming others help in avoiding delays? Blaming means you’re just looking at your part in the product development chain. You’re working optimally but everyone else sucks.

That attitude is not very helpful

In contrast: it will deepen the divides between departments and even increase delays due to less communication and trust. Accept that everyone is trying their best and assume that everyone is willing to contribute to the company’s joint success. With this basic mindset, you can approach other parties and work on increasing collaboration and trust while removing the delays.

Are you repeating yourself?

You know what I’m talking about: pushing new code to staging is a lot of manual work. And, if your testers need a fresh database dump, they wait for you to manually anonymize and import it. You’re sick of them requesting fresh code and data all the time because this manual labor keeps you from doing your real job. Manual steps in any process slow things down, create dependencies on the people who are able to do it, and are error prone.

You need to automate as much as you can

Setting up a new server should only be one command away. Loading and anonymizing a database dump? There’s a script for that. Rolling back a bogus change? Just another command and we’re green again. Continuous deployment and configuration management tools help you avoid those unnecessary delays.

This won’t work in my company

You might think that your environment is too complex to try any of the above. While it is true that a long history of silos and strong departmental boundaries make things harder, you’d be surprised to know everyone inside the other silos are striving towards the same goal: making your customers happy.

Approach others with respect and a positive attitude

By stopping the blame-game you can build trust. And by building trust you can increase collaboration and eventually speed things up dramatically.

You can start today

Changing your attitude is solely up to you. Approach your colleagues and discuss the process delays with a positive attitude. Together, try to find ways how to avoid delays by working on smaller items, doing less preparation work, and automating recurring, intensive manual labor. You’ll see that looking at a shared goal of avoiding delays gives everyone focus and energy.

By working on the whole product development process, you avoid getting stranded by a broken railway after rocketing over the longest leg of your journey at over 800 km/h.

We’ve teamed up with Packt Publishing to organize a giveaway of their new book “Continuous Delivery and DevOps: A Quickstart Guide” as a holiday gift to you our readers!

Four lucky winners stand a chance to win copies of this new book. Keep reading to find out how you can be one of them.

Continuous Delivery and DevOps

As the author puts it, the book is a “collection of suggestions based upon experience and observations.” It’s all about:

• Real world examples of how to go about implementing continuous delivery and DevOps
• Learning how continuous delivery and DevOps work together with agile tools
• An honest and open guide to consistently and quickly shipping quality software

Read more about this book and download the free Sample Chapter

How to Enter?

All you need to do is head over to the Continuous Delivery and DevOps book page and look through the product description of the book. Drop a line via the comments below this post to let us know what interests you the most about this book. It’s that simple.

Winners from the U.S. and Europe can either choose a physical copy of the book or the eBook. Users from other locales are limited to the eBook only.

Deadline

The contest will close December 31, 2012 at midnight Pacific Time (PST). Winners will be contacted by email, so be sure to use your real email address when you comment!

Don’t lose any time and head over to the Continuous Delivery and DevOps book page right now. We’re looking forward to your comments here on our site. Happy hunting and good luck!

Patrick Hoesly

While I’m collecting Devops Protocols which highlight healthy patterns in your organization, let’s take a quick look at the opposite: Devops anti-patterns. Would you be able to spot the warning signs when your team starts to slip in the wrong direction?

Committed is “Done”

It’s a good and trusted agile practice for a team to create a “Definition of Done”. It helps each team member understand exactly what’s necessary to truly finish a task. But watch out! Your Definition of Done might lead you down a thorny path or even cause short sightedness within teams. How could a Definition of Done do this? Let me explain.

Short sightedness within your team happens when each member only cares for their contribution to a task. They assume if they’ve finished, everything is fine. For a developer, this is thinking “committed is Done”; for a sysadmin “My script is running and I can understand it”. Sad to say it folks, but this isn’t Devops!

Every developer must think of the end user. Committing a piece of code is far from being done. It needs to work in all kinds of weird use cases. And it’s not only QA’s job to find all the bugs. Good developers want to ensure that the new features are not only coded, but tested and ultimately released to their users. Only then the task is really Done.

The same is true for sysadmins. Having a nice script on your own box is not enough. Every sysadmin needs to make sure it’s possible to re-create each part of the infrastructure at any time. When that slick, new script is under version control, written in a way others can understand and modify it, is their task really Done.

My Responsibility Ends Here

Another pattern I frequently see is the dreaded line in the sand: “My responsibility ends here”. A developer might think it’s not his job to see a feature through to release, a sysadmin might think that if the code breaks it’s not his problem.

They’re both wrong. To really create a high performing organization, able to rapidly deliver real customer value, individuals need to take broader responsibilities: everyone needs to care about getting valuable features into the hands of their users. No matter if there’s a code or operations problem, everyone should feel responsible and help find a fix. A team must pro-actively find ways to contribute to the solution of any release blocker, no matter where the fire is burning.

Take caution: if your team’s vision stops at seeing only themselves instead of the other parts of the organization needed to actually get their created value into users’ hands, then they have not taken full responsibility. Only taking complete responsibility will help foster the growth of a true Devops culture.

They should

A stronger form of the “my responsibility ends here” anti-pattern is the “us vs them” syndrome. It might start as soon as you move a few tables a bit farther apart. Suddenly, there’s a physical gap and people start thinking in “us vs them” terms. This thinking gets stronger the further people are separated. Different rooms, locations, and departments create islands of “us” and hinder collaboration.

If you start seeing “us vs them” emerging in your organization, fight it. Make sure people understand everyone’s contribution to the overall success of your organization and provide more interaction between these islands of “us” to improve communication. Only a feeling of belonging together will foster collaboration and lead to a Devops culture.

There are quite a few Devops anti-patterns: a flawed “Definition of Done, the stubborn “my responsibility ends here” or, even worse, “us vs them”. These are bad things you need to watch out for. If you see such patterns pop up around you, act immediately because they’re indicators that your team is on the path to short sightedness, sub-optimization, and blaming wars instead of growing a healthy Devops culture.

Have you seen such anti-patterns in your organization? Let us know your story in the comments below…