After DevOps Borat and BroOps it’s time for The DevOps Dudes!

Click through to see it full size

Other posts:

• DevOps is NOT a Job Description
• DevOps: Why Silos Suck And How To Break Them
• Do you have a DevOps Culture?

This is a guest post by Prasad Chaudhari, freelance java consultant. He was appointed as a project manager for the project mentioned below and played a role of ScrumMaster.

mattwilde66

Locally, we have six scrum teams: five software development teams, and one IT operations team. Each team has a sprint backlog, scrum board and burn-down chart. Automated tests and continuous integration are part of the daily business.

Our company has a universal understanding and acceptance of the Lean and Agile Value Chain, beginning with rough user concepts and ending with customer delivery. Without this basic building block, as the theory of constraints suggests, there will either be many unfinished stories in progress (i.e., because operations cannot make regular releases), or developers will starve for stories (because the business is unable to prioritize).

Last year, we decided to grow our software development capability by adding another scrum team in India. After a few iterations, reflections and process adaptations, the remote team became productive. We’ve been successfully running agile offshore for over a year now, and I’d like to share our key learnings.

Can offshore teams really be agile?

If it’s important to have a true understanding of agile and scrum on-site, it is absolutely critical to ensure the offshore team members have internalized the concepts of agile software development. Just knowing what scrum artifacts are, doesn’t necessarily mean that the team understands agile methodology. Every team member must comprehend the purpose behind each artifact. During the initial sprints, we explained at-lengths the purpose of each artifact and encouraged feedback sessions before each meeting to spot misunderstandings early.

The offshore team’s organization structure is the next thing to check. Usually, offshore service providers are structured hierarchically. During project resource staffing, we ensured that the organizational roles would not conflict with the project roles. Our employees are placed within one of three levels of organizational hierarchy. After being hired, new colleagues are briefed about the importance of their scrum role within a project, and how it does not affect their career progression within the company.

During every scrum meeting, we encourage all team members to speak openly about the project rather than only few member speaking for everybody. This frees up the more senior resources for more important architecture work and improves productivity.
Finally, the entire team cannot deliver quality if QA isn’t properly integrated. Testers must be empowered to ensure the delivery as per the definition of done. At the same time, they need to know that their job is not only to report and retest bugs, but deliver quality software together with the team.

Do offshore teams need extra scrum artifacts?

We use the standard scrum artifacts. Initially, we omitted the story preview meeting due to the scarcity of time available for POs, but we soon had to hold it in order to get better prepared stories for the “planning meeting 2″. We observed a very positive impact of stakeholder review on the productivity and morale of the team. Especially for long running (10 months) migration projects like ours, where the offshore team is not co-located with the business, the stakeholder review is a powerful artifact. Stakeholder interest for this meeting can be generated by asking them for what they would like to see/hear, adopting the presentation to their interests, or by encouraging them to click around the software.

Finally, if multiples teams are working on a shared code base, a scrum of scrums is very useful. Everyday, a member from each team visits this common stand up with representatives from the other teams. Each participant explains what their team is doing and if the team’s current activity might block other teams from working.

The Scrum Master Location Dilemma: On-site or offshore?

Simply put, the ScrumMaster should be located where the most impediments are. For us, most of the project impediments were at on-site. As our infrastructure and processes have been tuned over years to suit our local needs, they did not work well with the special requirements of working with an offshore team. Consider the remote availability of important support tools like Jira, our wiki, test databases, and maven repositories.

As I mentioned before, around 70% of development was done on-site and everybody was working on the same code base so it was critical to integrate all development activities. In our case, the ScrumMaster was on-site and his main task was to ensure that all external impediments like unavailable code repositories, broken builds, and lost database connections were addressed and resolved as soon as possible.

However, as the offshore team grew to 9 developers and 2 testers, the ScrumMaster needed to do this job in two places at once. To solve this dilemma, we created a hybrid system for fulfilling ScrumMaster responsibilities with a shared task list. Assigning these tasks to the offshore team, we made sure that the responsibilities are distinct and did not conflict with each other. This sharing of responsibilities will be expanded upon in the next section.

Offshore Planning Meetings slow and painful?

At project kickoff, “planning meeting 2″ was an incredible test of patience for the on-site participant. A new team, poor story quality and a complex domain made it difficult for offshore team members to make any sure estimates. The root cause of the problem was lack of well-groomed product backlog and story estimates in hours. Additionally, the offshore company management had set KPIs for the team that it should finish all stories to which the team commits during the planning meeting. This measure, taken to improve the productivity of the team, was surely in the back developers’ heads while estimating stories. We solved this problem in multiple ways. At first, by enforcing story preview meetings and making sure the stories are well defined and small. Although it helped considerably, meeting times were still not reasonable.

Initially, we did planning meeting 2 immediately after the planning meeting 1, but later we introduced a gap of few hours between these two meetings. In this gap, the offshore team was given a chance to research, brainstorm and come up with the design. This reduced the time of planning meeting-2, and everybody in the offshore team had a chance to actively take part in the discussion.
We split the offshore team (9 developer and 2 testers) in half and appointed a team lead. To avoid the trap of hierarchy, we called them “story responsible” and rotated this role.

Now, planning meeting 1 is done with the entire team, but planning meeting 2 (the detail planning) is done with two teams. This has removed the wasteful pain of an entire offshore team trying (and failing) to plan all stories. With a team lead sharing the ScrumMaster responsibilities, we have also achieved linear scalability. The two team leads take over the moderation of meetings, retrospectives and solving offshore impediments.

Are you communicating well?

As we all know, communication is key to any process. Making use of the predefined, scrum artifacts ensures that communication takes place, but it does not ensure the quality of communication. Quality of communication is directly proportional to the quality of interpretation by the receivers. Ambiguous communication is differently interpreted by the individual project team members. We continuously improved our quality of communication by trying different communication channels and feedback loops.

After months of experimenting, we settled upon these three communication channels: Skype for video, telephone line for voice, and “live meeting” for desktop sharing. Having an uninterrupted voice connection was vital; especially when 11 people are all taking part in a meeting. Hence, we decided to use a land-line for voice communication instead of Skype or Google talk.

Although telephone calls are more effective than emails, certain documents (like guidelines) need to be written down. Most of the on-site communication was implicit, based on situational awareness and simple co-location – definitely not enough for an offshore team to start implementation.

Therefore, we discussed the requirements together with the offshore team and asked them to write the final specifications. The on-site team reviewed these during development and after completion to suggest improvements. To provide a strict framework for the offshore team, we wrote the guidelines for clean-code, pre-commit checklists, and operational wiki notes.

Creating Situational Awareness

One of the powerful tools scrum provides is the storyboard. Of course, for distributed teams, use of common physical board is not possible, so we use Jira with the excellent Greenhopper plug-in. As all 6 teams are working on the same code base, we use Teamcity for continuous integration and keep the build green by using pre-commit hooks. Individual developer commits are staged on the integration server, compiled with the latest revision of code, and, finally, run against all automated tests. Only after passing all of these quality gates, is the commit pushed to the source code repository for the rest of the team.

To summarize, adopting agile software development processes was instrumental to the success of this project. Developing working software in each sprint provided us, as well as company management, with confidence in the early phase of the project, and retrospectives gave valuable input for fine-tuning the process.

Having a fully functional PO role would have definitely helped in further improving the team’s productivity. But, we successfully achieved linear scalability by splitting offshore team into two and letting both team leads share some of the ScrumMaster responsibilities. For the next project, we are working on a more well-groomed product backlog, and better story point estimations.

Looking back, I believe the most important learning was continually improving every aspect of our development. After all, being agile means ‘readiness for change’, doesn’t it?

Other posts:

• Waterfall, SCRUM and Lean Software Development simulation as teaching platform
• How “Good to Great” applies to agile software development
• State of Development: Annual Address on How We Ship Software

Wonderlane

The DevOps hype produces some strange effects. Not only do tool vendors try to jump on the DevOps band wagon by declaring their products “DevOps inside” or listing DevOps as a feature, but companies start to look for a “DevOp” in their job ads. Don’t be misled! Here’s what DevOps is really about:

DevOps Is About Culture

The fundamental basis for successful DevOps is a culture of trust and a feeling of fellowship. Everything starts with how people perceive each other: Is it an “us vs them” culture or is it a “we”-culture? I don’t see any job description in here.

DevOps Is About Automation

Let’s look into automation. A lot of the advantages DevOps promises lies in the right use of automation tools. Automation removes variance from your processes and minimizes human error. While you’ll definitely need people with automation skills and experiences, this is not enough. Automation only – without the right culture – will not provide the benefits you hoped for.

DevOps Is About Measuring

Then let’s examine measuring. While measuring is a critical, mandatory practice for improving processes, it’s not really a job description. Every employee should formulate hypotheses, run experiments, and validate or scrap her ideas. No magic sauce here either.

DevOps Is About Sharing

What about sharing? I’m sorry to say that while the DevOps movement is largely driven by sharing ideas, problems, and tools, this isn’t really a good job description. It is an attitude more than a task.

If you run through the points, which really define DevOps you’ll see for yourself how strange it is to try and hire a “DevOp”. DevOps is not a job – DevOps is Culture, Automation, Measurement, and Sharing (CAMS).

Other posts:

• Do you have a DevOps Culture?
• 2011: Time to Escape the DevOps Echo Chamber
• DevOps: Why Silos Suck And How To Break Them

| spoon |

Dev and Ops are separate departments

Organizations typically divide their teams by type of work. They create development departments for writing software. And, because running software quickly and stably requires a different set of specialization, they create a separate operations department. This division of labor simplifies management because each department shares certain characteristics and needs.

Separated departments have conflicting goals

Each department defines goals along division of labor lines. The development department may be measured by the speed of creating new features, while operations is judged by uptime and response time. Unfortunately, operations is considered successful with stable, unchanging metrics, but development is only applauded when lots of things change. Conflict is baked into this system making collaboration unlikely.

Dev and Ops sub-optimize to meet local goals

Instead of optimizing the whole application lifecycle, both Dev and Ops improve their processes to meet their respective goals. Developers do everything to speed up the creation of new features by adopting agile methodologies. Sysadmins do everything to maintain stability and enhance performance by applying ITIL which equates change to risk – something to be avoided at all costs.

In an even crueler twist of fate, when each of these departments are operating at maximum efficiency, the product lifecycle suffers the most. Development is continuously pounding out new features against the walls of Operations which, in turn, has all kinds of defense mechanisms in place to fight change. A huge war is about to break out, and the only way to avoid it is by aligning the goals of these two departments.

Other posts:

• DevOps: Why Silos Suck And How To Break Them
• The 5 Goals Of Agile And DevOps
• DevOps Q&A with Kevin Parker

Claudio Gennari

In my case, the core project team is 100% committed, but there are a few external folks from support and QA who are involved in many other projects. From their viewpoint, our project is just another task on their ever-growing checklist of product support. They don’t care about new user features – they want the bugs fixed. Because the project team only gives points to new features, the support colleagues feel left out and forgotten. Their job is seemingly made harder by our “overlooking” bug fixing.

Let’s be clear – we do a lot of bug fixing – actually, way too much! As we have no automated tests, we get the disastrous effect that fixing one bug causes three others. Roughly 50% of our sprint stories are bug fixes.

A cross-dysfunctional team like we have will never be able to come together and agree on the importance of regularly delivering customer value and unit-testing. Their loyalties lie with different departments and organizations. To get a truly committed, cross-functional team everybody needs to pull together in the same direction, to have a common understanding of the teams goals and feel 100% empowered in making the decisions to make their vision reality.

As for how we can increase our velocity, the answer is quite clear: start writing tests. At the very minimum, stop making anti-fixes!

Other posts:

• Self-brewed complexity is evil – fight it!
• 6 Bad Ways of Conveying Urgent Tasks (And How to Fight Them)
• Estimation of User Stories With Story Points as Abstract Size Measure

Seattle Municipal Archives

It’s amazing. Talking to a bunch of fellow CTOs I heard a lot of them saying: “We introduced Scrum and it works really well” and “we’re too slow to bring new features to our customers”. This piqued my curiosity. Scrum is supposed to speed up feature delivery through short iterations. How can an organization claim to run Scrum successfully but not deliver customer value fast?

The Curse Of Sub-Optimization

For me, that sounds like a typical case of sub-optimization. You optimize only one part of a value stream but not others. While that one part (e.g. development) is getting faster and faster other parts of the system are not able to keep up the new pace. The development team will either run low on User Stories or it will swamp QA or Operations with features to be released. In the worst case, you end up slowing down the whole feature creation process instead of speeding it up.

The Pipeline And The Bottleneck

I like picturing the software development process as a pipeline. From the initial idea to production release a feature needs to flow through the organization. You can picture the capacity of each step in that flow as the width of a pipeline. The higher the capacity of one part of the organization the wider the pipe. The bottleneck of the pipeline is the part of the organization with the lowest capacity to deliver features. You can easily identify it by visualizing your process e.g. on a Kanban board and looking where the most Kanban cards get stuck. If you optimize an area in your process, which is not the bottleneck, you’ll end up with sub-optimization. Only optimizing the bottleneck will really speed up overall delivery.

Scrum Moves The Bottleneck

If development used to be your first bottleneck and you introduce Scrum successfully you’ll see some speedup in delivery. But it might be way less than expected. Other parts, like QA or Product Management, may have a capacity which is only slightly higher than that of development. If you keep optimizing development, you won’t achieve any more speedup. Instead, you need to focus on the next bottleneck and optimize it. Only by constantly looking for the current bottleneck and optimizing it can you constantly speed up your delivery.

Other posts:

• Scrum vs Continuous Deployment or why Scrum falls short for web applications
• How Scrum Induced Sub-Optimization Kills Productivity (And How To Fix It With Kanban)
• Scrum What? New Community Edited Q&A Site About Agile, Lean, Kanban and Scurm

Jo Naylo

You’ve most probably been there: To win that one ueber-important client, your friendly sales rep sells the farm and his grandmother (well actually he sells features, which he invents right in front of the client to make sure to get the deal, but the effect is nearly the same). And not only does he sell „the grand new feature in the sky“, but he even guarantees a delivery date – and all without speaking a word to the tech guys. Sweet, huh? He is a hero. He made that huge deal and everyone is patting his shoulder. Well, besides you probably. And everyone else who now has to deliver a completely bloated feature in a totally unrealistic time frame. And, of course, there’s absolutely no room for negotiations. In fact, you’re not even allowed to talk to the client to ask a question about his requirements because the sales guy doesn’t want to make a „bad impression“.

Non-negotiable Features Create Technical Debt

You know what happens next. The developers scramble to their feet and churn out something, which, at first glance, can be considered as the feature the client ordered. You don’t have time for adapting the architecture to the new needs. And, of course, automated tests, proper feedback, and a useable UI go right out the window. You might be able to pull off that crazy delivery date, but at what cost? You’re left now with a bloated beast of software put together in such a rush that no one dares touch once it’s released because the chances of breaking something are significant. You dug your technical grave, but the rest of the company is celebrating a huge win! No one outside the tech department noticed how huge the technical debt you’ve just taken in order to deliver.

No-one Profits From Non-negotiable Features

If features are pre-sold without any option to negotiate what’s important and what may be left out, you inevitably end up with too much complexity. Sales representatives often feel pressure to promise too much to a client. They want to close a deal, but they’re not able to see the technical implications. They raise very high expectations and often fix them in the contract which removes any way to iterate and develop your product until it’s actually useful for your client. Instead, the contract is based on a rough, but very challenging scenario, which is impossible to deliver and has very little chance to fit the client’s needs once it is released. Such pre-sold features not only tie your hands, but the client is also not able to change what he needs over time. Both parties agreed to the contract and a rigid change management process is installed to make change as painful (and therefore infrequent) as possible.

Other posts:

• A Kanban Board for Features
• Continuous Integration Helps Find and Kill Bugs
• Waterfall, SCRUM and Lean Software Development simulation as teaching platform

GoTRISI

No clear picture of your dream candidate

Usually, people slap together a list of things they need a candidate to know or to have experience in. But this isn’t enough. Too many companies are already searching similar profiles on that level of detail. Imagine your dream candidate: What is his current situation? What is he doing right now? What are his fears and dreams? Only if you zoom in and create a very vivid and detailed picture of your dream candidate, will you be able to address the right people.

No compelling message

A job ad is like a sales letter. It needs to address the problem you solve for your candidate (find a great job) and it needs to show why your job is the best solution. Additionally, it needs to address objections the candidates might have reading your job ad (“will they keep my inquiry confidential?”, “will they offer fair payment?”, etc). Furthermore, it helps to add some testimonials to your job ad: Let your team mates describe how switching jobs was for them. And, last but not least, you need to create a compelling reason why your job is unique and not just like any other engineering job out there.

Searching the wrong places

The best job ad is worthless if it doesn’t reach the desired candidates. Instead of just posting it to general purpose job sites like monster.com, you have to chase down your dream candidates and find the places they hang out. Failing to do so means getting flooded by the wrong candidates and wasting a lot of time and money. If you’re looking for a ruby expert, who regularly shares his thoughts and code, you might find them on twitter, github, or stackoverflow.

Poor Handling of Applications

Too much can go wrong in the final stages of the hiring process. The best way to lose great candidates is waiting too long. Waiting too long to invite them for an interview, waiting too long to give them feedback, waiting too long to make an offer. When you start the recruiting process you need to define the exact procedures for following up with candidates. Who needs to decide what? How fast can they make those decisions, etc. If you communicate a clear process with well defined timing, chances are your candidate will be engaged and stay tuned.

Failure to close the deal

After going through all the necessary interviews, you’ve lined up a dream candidate for hiring. This is a critical step in the process. You need to make a compelling offer and dissolve all objections the candidate might have about signing up with you. And it’s time again to re-emphasize your uniqueness to stop the candidate shopping around for comparable offers. Only if the candidate knows everything about your company which helps them make a positive decision, will you be able to close the deal.

What are your worst experiences when trying to hire someone or when trying to join a company? Please share your horror stories in the comments.

No related posts.

rmlowe

No matter how small and insignificant you might be, you can still be a catalyst of powerful change. An idea may give somebody the “Aha!” moment which, in a heartbeat, literally changes their entire outlook on life. They impart this change upon others and the process reverberates outwards – babbling brooks forming streams.

Telling a developer about the importance of automated testing is a mandatory, minimum requirement of being a software development manager. Kinda like grade school. You learn how to read, write and do basic arithmetic. Maybe you can even remember a favorite teacher, but the rest is a blank.

Showing a developer the ease of automated testing might get a spark started – TDD can become infectious if done right. You’re implementing learned knowledge (and, more importantly, how to cut corners and save precious time for other activities). And, this, unfortunately, is about the highest level most folks ever reach. They survived the manager from hell that forced them to write tests and improve code coverage without really understanding the underlying principles.

You must convince developers that code quality is their responsibility. Such an idea is on par with a university course covering human ethics and engineering. This is the tiny rock particle I mentioned at the beginning. And getting it moved will be one of many causes of positive and lasting change.

Just like telling a three year old “No!” for the first time, resistance will be swift and firm. Most developers burst out in a tantrum of foot stomping and howling. They don’t have the time much less the training to write tests. It’s not their job. Remain firm and show the benefits of moving quality control upstream. Toddlers know, deep in their hearts, the difference between right and wrong. Once you convince them they have the time, developers will too.

Next up on the list – telling system administrators about the importance of version control!

Other posts:

• A Luxury Problem: How Emerging Iterations Eat Team Commitment For Breakfast

rmlowe

Maybe you read it long ago, or it’s been on your “to read” list for years. Or maybe you’ve never heard of it: The book “Good to Great” by James C. Collins. It describes how companies move from being average to great and how they can fail to make the transition. So, what does all this have to do with agile?

The book describes three main stages:

• Disciplined People
• Disciplined Thought
• Disciplined Action

Disciplined People

In this section, Jim Collins talks about “Level 5 Leadership”:

Level 5 leaders are ambitious first and foremost for the cause, the organization, the work –not themselves– and they have the fierce resolve to do whatever it takes to make good on that ambition.

I usually refer to such leaders as “Servant Leaders”. These are the people who need to be in charge of your organizations. Managers only interested in micro-management and power-games, should not be put into critical roles!

In addition to “Level 5 Leadership”, Jim Collins talks about “First Who…Then What”:

Those who build great organizations make sure they have the right people on the bus, the wrong people off the bus, and the right people in the key seats before they figure out where to drive the bus.

I couldn’t agree more: It’s of paramount importance to make sure you have the right people on the team. It’s better to wait (and take on the extra work) while searching for the right one rather than taking just anybody. And, I can hardly think of anything more toxic than having the wrong people. It sucks the life and morale from your whole team. Think of the people you want to work with first. Everything else comes second.

Disciplined Thought

Disciplined thought, as described by Jim Collins, is extremely helpful in focusing on the right thing. He talks about the so called “Stockdale Paradox”:

Retain unwavering faith that you can and will prevail in the end,
regardless of the difficulties, AND AT THE SAME TIME have the discipline to confront the most brutal facts of your current
reality, whatever they might be

I’ve seen this as one of the most important traits of any successful team. Only if you’re always brutally honest with yourself, can you start changing things for the better. But, you’ve got to trust in your abilities to change things for the better – otherwise you despair. Retrospective meetings and root cause analyses are perfect places to “confront the brutal facts”.

When talking about “The Hedgehog Concepts”, things get a little more complicated.

The Hedgehog Concept is an operating model that reflects understanding of three
intersecting circles: what you can be the best in the world at, what you are deeply passionate about, and what best drives
your economic or resource engine

This is very important, but very advanced. You need to have a very self-conscious organization or team to be able to touch these underlying truths. But, knowing about them and working toward clarity in those “three circles” never hurts.

Disciplined Action

The first step of disciplined action is creating a “Culture of Discipline”:

Disciplined people who engage in disciplined thought and who take disciplined action – operating with freedom within a framework of responsibilities

This perfectly describes a very important cornerstone of any agile process: Self organizing teams and disciplined engineering: Test automation, and easy to change architectures. There can be no success without a “Culture of Discipline”!

In the last section, Jim Collins talks about “The Flywheel”. The path to excellence is not opened by an epiphany but by constantly “turning the flywheel” until you have so much speed that it catapults you into regions of excellence you’ve never been able to dream of. This is really one thing I experience time after time when introducing agile practices in any team or organization: At the beginning it seems you’re only baby stepping and hardly making progress. But, as soon as a team learns the importance of ownership, clean code, and real team work, they reach a speed and quality with their results that is astounding. No one would have thought such great things to be within reach of the team.

Reading from “Good to Great” really opened my eyes in many areas. A lot of points resonate with my experience in working together with high performing agile teams. And mastering the hedgehog concept is a deeply powerful thing to guide your organization. Have you read the book? What are your thoughts? Please share them with us in the comments.

If you want to assess your organization in terms of the above concepts you can use Jim Collins’ Diagnostic Tool (a PDF document describing the concepts and asking you a few questions for self-assessing your current situation)

Other posts:

• Optimizing Offshore Software Development with Agile
• Waterfall, SCRUM and Lean Software Development simulation as teaching platform
• State of Development: Annual Address on How We Ship Software