Agile Making Progress

Back on the Front Line of Agility...

2012-03-06T04:33:00.001-08:00

Sorry that it has been a while since I posted here - my professional life has changed considerably since I last wrote!

Last Fall I join an exciting startup, VoltDB, leaving behind a "Waterfall to Agile" transformation environment, diving head first into the front lines of a pure Agile start-up environment.

It's been a pretty exciting 6 months working with a great team of engineers. We're delivering a high velocity, scale-out NewSQL relational database. This product fits at the fire hose intake edge of Big Data and is capable of ingesting hundreds of thousands of transactions per second on commodity hardware. Low latency, high velocity. Lots of challenges, both technical and process-wise.

Here's an overview of the VoltDB Development environment:

Open Source Database. Because it is open source, anyone can see the our code (see our Git repository at https://github.com/organizations/VoltDB).
It's a database. It has to work, can't corrupt or lose data. Quality is paramount.
We have Continuous Integration system running a large suite of automated tests.
VoltDB is in production for a significant number of customers. There's no dedicated Support team; the Engineering Team provides 24x7 support.
We have transparent development: our Product Backlog and Sprint Backlog is hosted (publicly) in JIRA. We use Greenhopper to plan and track iterations.
3 week iteration. The product is released almost every iteration! The Team truly produces a potentially shippable product increment most iterations.

Of course, like every Agile team, we have our share of challenges. Some of the challenges we experience include:

Dealing with unplanned customer support issues as well as pre- and post-sales assistance.
Dealing with technical debt.
Story sizing: specifically defining small enough slices of potentially shippable functionality.
Scaling the team: we're growing fast!

We've adopting Scrum to help us meet these challenges. I'm a firm believer in Agile software development, specifically an Inspect and Adapt framework like Scrum. Because we're a start-up with a product smack in the middle of a rapidly growing, and evolving, Big Data market, product requirements can change often, sometimes daily. Scrum provides us the framework to adapt to changing business conditions while still regularly producing business value.

It's great being back on the front line of Agility. I hope to find time to blog about these challenges as well as other topics in the coming months, time permitting.

Scaling Scrum: The Chief Product Owner

2011-09-13T08:18:00.000-07:00

We’ve begun our next phase of Agile transformation - scaling Scrum from single team efforts to larger multi-team projects within our organization. As part of this phase, we’ll be building and delivering multi-product enterprise-ready offerings, requiring the coordination of many Scrum teams across many time-zones.

Not unexpectedly, as we’ve delivered the first few product iterations from these larger efforts we’re faced with new challenges. Among the biggest are coordinating backlogs between teams and resolving dependencies and integration issues. Both issues are requiring a lot of additional time and effort, over and above the team-level Product Owner and Scrum Master efforts.

In his book about scaling Scrum, titled The Enterprise and Scrum, Ken Schwaber states “The higher the level is, the harder the Product Owner’s and Scrum Master’s job is.” This is an important point to recognize and deal with: as you expand the size and scope of a product, as you “scale up”, more work is required for those holding the key roles in Scrum. Though this is a huge challenge, there are common patterns that can be used to help. One is the Scrum of Scrums, where one or more representatives from each Scrum team meet daily to coordinate Sprint work and Sprint dependencies. Another pattern, one that I consider extremely important when scaling, is a Chief Product Owner.

The Chief Product Owner is a new unofficial (to Scrum) “role” that can greatly help with scaling Scrum. In Scrum-speak, this role is the Product Owner of the whole product. The Chief Product Owner is a person who is the single point of accountability for the success or failure of the complete project. This person owns the overall single product-level Backlog, and provides guidance to the group of product owners on the Scrum teams that comprise the complete effort. The chief Product Owner uses this backlog to coordinate the work, likely in the form of sub-backlogs, between the sub-teams, through each individual team’s Product Owner.

It is important to note that this role is filled by an individual, not a committee. Without a designated person ultimately responsible for the success or failure of the effort, there is a high risk of “death by committee”. By identifying a Chief Product Owner you can avoid this and have one final arbiter to order work and accepting results, as Scrum dictates for individual teams.

The Product Owner role is a key role on individual Scrum Teams. When scaling Scrum, this role takes on even more importance - the success of the entire product relies on deep understanding of the overall product backlog and careful and coordinated roll out of work to individual Scrum teams. Like the Product Owner role on an individual Scrum team, the Chief Product Owner is a full time role, one that requires a full time focus and commitment in order to be successful.

Scrum: It’s a full-time job.

2011-08-29T09:57:00.000-07:00

“An adequate ScrumMaster can handle two or three teams at a time...A great ScrumMaster can handle one team at a time.”

-Michael James

One of the lessons we learned early in our Scrum roll-out was that the Product Owner and Scrum Master roles required much more involvement than originally thought. Coming from a Waterfall development environment, where releases were long in duration and milestones spread out, people in the organization were used to having key roles on multiple projects, context switching as needed.   Juggling responsibilities in a Waterfall environment seemed to work well, or at least acceptable. Our Product Managers, for example, could own several product lines, including associated customer meetings and field interactions, and easily satisfy occasional project touch-points such as requirements definition (at the beginning of the release) and Beta and FCS duties (at the end of the release). These project touch-points required their full-time involvement periodically, a few times a year, making it relatively easy to context-switch periodically and satisfy the needs of each effort.

With Scrum, however, things turned out a bit different. Though we “took the training” and “read the literature” which repeatedly said the roles of Scrum Master and Product Owner were full time jobs, we weren’t true believers. Our “muscle memory” way of working had everyone juggling multiple roles and responsibilities, it was the way we were used to working. During the course of our Agile Pilot projects, however, we got convinced pretty quickly that Scrum roles could not be part-time.

Scrum is more intensive than waterfall development.

With frequent “potentially shippable product increments” created every 2-4 weeks, there’s essentially a full product release at least once a month if not twice.   During our Agile pilots, our Product Managers, filling the role of Product Owners, soon realized that creating and maintaining a Product Backlog required a significant amount of time. Every few weeks a new set of fully-defined “product requirements” need to be ready for the team for the next Sprint. Product Owners found it challenging to satisfying their Scrum PO role and concurrently complete other project work. Our Scrum Masters, too, found themselves booked full-time, facilitating team interactions, producing Sprint Burn Down charts, working impediments, and delivering frequent product releases at regular Sprint Review. Team members, often with other-project responsibilities, found themselves spending more time delivering on Sprint Goals, often at the expense of their other responsibilities.

As we scale Scrum within the organization, we’ve (obviously!) needed to identify new Scrum Masters and Product Owners. We’ve held fast to the one project per Scrum Master recommendation. Identifying new dedicated Scrum Masters has been fairly easy - our Pilot teams was that the efforts produced several Scrum Master candidates, a nice side-benefit!   Finding full-time Product Owners for each team was more challenging. Because we’re applying Scrum to large multi-team projects, our mapping of Product Manager positions to Product Owners simply didn’t scale.    Our solution was to look to our lead engineers. Most of our new teams have technical Product Owners, usually Software Architects, and they are dedicated full-time to the team. Product Managers are involved at either a co-Product Owner (in an outward facing role) or at a higher level in the project effort, via a Product Council-type role that involves them as stakeholders.

I expect our organization to fine tune (via inspect and adapt) these non-Scrum roles as we mature in our Agility. But regardless of the organizational “noise” around the Scrum Teams, I believe that participation at the Scrum Team-level will continue to require full-time effort from all members in order to be successful.

Why Agile?

2011-08-09T08:09:00.000-07:00

We’re at what I perceive to be a major “transition point” in our Agile transformation. Most of our Agile pilot teams and projects have successfully completed and we are rolling out internal Scrum training throughout the Product Group. We’re on the cusp of taking a big step forward with Scrum adoption within our 500 person organization. It’s at transition points like this when I find myself reevaluating our direction: “Why are we doing this? Are we ready for this next big step?” Our existing processes do, in fact, lead to shipping product and generating revenue, so: “Why Agile?”

Our original goal, defined we began this transformation, still rings true:

The goal is not to release things faster for the sake of it. It is to achieve nimbleness and agility to deliver on the company’s strategy as it moves forward and/or tactically changes. It is about creating more business opportunities for ourselves rather than being cased into a model that requires our resource commitment for a time window that is longer than what the market we’re competing in requires.

Specifically, the software development group’s reason for moving from Waterfall to Agile is to achieve nimbleness and agility to deliver on the company’s strategy as it moves forward and/or tactically changes. We want the ability to efficiently deliver strategic products to market in a timely manner.

Today we have challenges delivering on this goal. As a mature software organizations our development practices have been hardened over many years of delivering software to tens of thousands of customers. Our processes have been tempered with the successes (and often, the failures) of the past. Because of this, it’s difficult to change the current way of doing things. This is an understandable, and completely human, reaction - but it can dangerous over the long-term. Fortifying the walls, via policies and procedures as well as organizational divisions, has the result of business being less reactive to change, less agile, which ultimately impacts corporate revenue. New software releases often take longer to get out the door and may not fully meet customer needs. Entering new markets in a timely manner becomes difficult, allowing younger more nimble companies a market advantage. It is for these reasons that we’ve embarked on our development process transformation.

As part of our migration from Waterfall to Agile software development, we need to build a new way of working, including a new corporate culture: one that is nimble and responsive to ever changing business conditions. A culture that values responding to change over following a plan. One that regularly inspects and adapts, and becomes better over time. Scrum, with its “inspect and adapt” framework, is one of the catalysts that can greatly assist us in this journey.

Technical Debt

2011-07-19T07:22:00.000-07:00

Technical Debt is one of my favorite work-related phrases because it succinctly and metaphorically sums up the obligation we incur when we cut corners during software development. The phrase was coined by Ward Cunningham, the developer of the first wiki and, incidentally, a major contributor to Extreme Programming Agile methodology.

The concept of technical debt can be thought of as the accumulation of product risk introduced by any number of poor development practices. Examples include poorly designed code, loose development practices, and, in Scrum terms, the delivery of “undone work”. The payment terms on this debt can be onerous. Usually the cost is time or productivity, sometimes both. Dealing with this debt during the product release can increase the work required before the product can ship, often delaying the release. Or, if you decide to postpone payment until after the release, the cost may be customer-discovered defects which can cause unexpected maintenance escalations and a higher ongoing product maintenance burden, ultimately reducing productivity and velocity for future development.

Regardless, like credit card interest or the National Debt, at some point in the future the bill comes due - someone is going to have to pay off the debt.

For new teams adopting Scrum, Technical Debt can occur without the team being fully aware that it is happening. For example, teams with a loose definition of Done can introduce technical debt. Stories may be accepted by the Product owner without having code reviews, minimal automated testing, or no code coverage or memory leak verification. Any of which could leave unexposed defects in a seemingly “done” feature. Another possible cause of debt can be caused by user stories that not properly decomposed such that Sprint backlogs do not yield a potentially shippable product increment at the end of a Sprint. In this case, more work, either known or unknown, is created for “later” in the release.

As a Product Owner, you should keep track of your accumulating debt. If the debt is significant, I recommend putting it in your Product Backlog, to be prioritized appropriately. One way that I have found to identify and track any technical debt your team is accruing is to plan periodic “releases” to customers. Product early releases, in the form of early access programs or Technical Previews, can not only provide valuable customer input that can feed into your backlog, but it also keep the Scrum team in the practice of delivering potentially shippable code. When engineers realize their code is going to be used by actual customers, often “forgotten” undone work can suddenly appear.

Note that not all debt is “bad” - there are often valid reasons for accumulating technical debt. Perhaps the underlying architecture has some scalability constraints, but getting the product to market, in other words, harvesting “business value” ($, customers), is more important to the business. The key is for the Product Owner and Team to explicitly know that this debt was incurred, and accept that you may have to pay in the future be willing to accept that obligation.

The Product Owner should be aware at all times of work that is needed to release the software, including the technical debt that is accruing. With a complete and publicly visible Product Backlog, there should be no surprises. Keep an eye on your debt - make sure you know how much debt interest is accumulating...

Scrum Cadence

2011-06-29T06:06:00.000-07:00

It’s unfortunate that the the Scrum product iteration increment is called a Sprint, implying that each increment is delivered by the team in a mad rush, full throttle. The image I have in my head is a 100 yard dash, everyone running as fast as they can to reach the finish line. Developing software is more like running a marathon than it is a back-to-back series of 100 yard dashes. In order to successfully complete a marathon you have to find a one mile pace you are comfortable running at, and keep running at that speed for 26 miles. Sprinting hard for the first several miles often means you are walking (or even crawling) the last mile, if you even make it that far.

Me, still running at mile 26, in the 1998 Boston Marathon

One of the challenges many new Scrum teams face is developing a sustainable pace, sometimes called cadence, over multiple sprints over the course of a project.

New to Scrum, teams often struggle to find a sustainable cadence for delivering potentially shippable product increments. What I’ve observed happens is this: Early in a new Scrum project the teams is (I hope) excited to adopt a new way of working. They understand the Scrum principles and the time-boxed Sprint iteration, and more importantly, they feel a strong commitment to succeed on their newly committed Sprint deliverables within the first few Sprints. After all, at the end of the Sprint they will be proudly demonstrating their work to their stakeholders, which usually includes there boss and department head. However, since everyone is new to this different way of working, it is inevitable that certain work, usually that requiring specialized skills such as QA and documentation writing skills, ends up being delivered late in the game. Undeterred, the team works extra hours, heroically delivering the Sprint goals. This extra effort delivers a much needed early success, builds organization confidence, and can kick-start viral adoption. However, this type heroic effort doesn’t scale - after 2 or 3 Sprints the team starts feeling burned out and may fail to deliver Sprint-committed work.

In an ideal situation, the Team inspects and adapts, and, over the course of several sprints, build a sustainable cadence. However, many teams will continue to struggle, and can fall back to the “old way of doing things”.

While every team, project and environment is different, here are some points to consider to help get to a sustainable cadence:

Work, as a full team, on only one story at a time. Make sure that story is DONE before starting the next story. In other words, implement stories sequentially.
Avoid picking stories targeted for one individual due to their specialized skills. Make sure the whole team is involved in delivering each story.
Develop an “in it together” feeling among the team. If you aren’t feeling this, your team isn’t operating as a true TEAM.
Commit to less stories to make sure your Sprint backlog really is DONE at the end of a Sprint.
Become more of a generalist. Scrum certainly favors “jack of all trades” type of developers. Coming from a Waterfall environment, where there are separate groups for QA and documentation, work can fall to those with that domain specialization. As a team member, don’t look at work as “that person’s”, view it as a team deliverable. Learn a new skill to help a team member (and thus the Team) out, be it coding, QA or documentation.
Be honest and innovative during your Sprint Retrospectives. It’s your chance to change how you work together to deliver software.
Get additional training or coaching. Read Scrum blogs and ask questions on Agile forums. Likely you are not the first team to run into the issues you are experiencing.
If you are manager reading this, realize that team cadence, a predictable velocity, is key to delivering on corporate objectives. Work hard to remove team obstacles and allow them to focus and succeed. This is the true value of a manager in an Agile world.

Developing a sustainable cadence is one of the keys to the success of the Scrum Team and project effort. Remember, running a marathon full throttle is unsustainable, unless, of course, you are an ultra-elite runner. I know I’m not one.

Inspect and Adapt

2011-06-08T05:48:00.000-07:00

“SCRUM assumes that the systems development process is an unpredictable, complicated process that can only be roughly described as an overall progression.”

-Ken Schwaber, Scrum Development Process, 1995

One of the things that I like about Scrum is that its creators defined it knowing that no development process is perfect. The Scrum Guide does not tell us how to build software, rather, it openly acknowledges that building software is complicated and unpredictable. This may seem odd, with Scrum being thought of as a development process (it’s not). In a recent post to the Yahoo! Scrum Development group, Ken Schwaber stated that “we purposefully constructed Scrum with a lot of holes in which a person or team or organization has to continually figure out the best thing to do.”

So how does a Scrum team “continually figure out” how to do the best thing?   To facilitate iterative and ongoing improvement, Scrum specifies that each Scrum Team hold a Sprint Retrospectives. Further, these retrospectives must happen frequently, occurring immediately following each Sprint Review.

It is often overlooked when starting out with Scrum, but I believe the Sprint Retrospective is the most important meetings a Scrum team participates in. While the Sprint Review is a valuable “public” meeting where working product is demonstrated to stakeholders and customers, the Sprint Retrospective is a “private” team meeting, allowing the team to safely discuss what’s working and want could be improved with regards to how they are delivering their work. In the Sprint Retrospective, anything that relates to how the team operates can be discussed and possibly tuned or improved. Up for debate could be how the team communicates, technology issues, done-ness criteria, or even choice of tools, as was recently discussed for one of our new Scrum teams in my organization (They reverted back to whiteboard task charts and spreadsheet burndown charts!).

What this ultimately means is that each Scrum team is evaluating, refining, and improving how they work together and build software every 2 to 4 weeks. Almost continuously improving throughout the life of the project! Over a 12 month project there can be upwards of 24 opportunities to “do things better” and improve how the software is built and delivered.   Compare that to our our Waterfall-like development model where we generally conduct formal retrospectives twice, once after Beta and once after we release.   In our current model, we, at best, have one predefined opportunity to improve our practices during the release, at Beta when there is usually no more than 3-4 months of (QA and bug fixing time) time remaining before the software is released. Process improvement is clearly not highly integrated into our Waterfall process.

With frequent inspection and adaptation we can become more efficient at building and delivering high quality software that meets or even exceeds customer expectation. So my advice is: take full advantage of your Sprint Retrospectives.   As a Scrum team member, it’s your (frequent!) opportunity to make a difference...

kaizen (kaɪˈzɛn) - n a philosophy of continuous improvement

of working practices that underlies total quality management

and just-in-time business techniques [Dictionary.com]

Cycling and Scrum

2011-05-25T09:45:00.000-07:00

Every year I join a group of friends and ride a 100K charity bike ride. This ride takes about 4 hours and has turned into a fun yearly tradition. This year, as I was battling my way up the hills of Chilmark (the ride covers most of Martha’s Vineyard), it occurred to me that there were a lot of similarities between the cycle ride I was a part of, and a Scrum team.

Some of the similarities I came up with were as follows:

Once you know the basics, you can ride a bike a long distance, or participate on a Scrum team. However, having solid training for each makes doing them much easier and more efficient.

Developing a regular cadence is valued in both Scrum and cycling. Going to hard and fast early on can quickly burn you out, making it hard to finish the job.

Buying a Scrum tool doesn’t make you a better Agile organization, just like buying a new carbon-fiber bike doesn’t make you a better, faster rider.

I’m a long distance runner, and not a great cycler. However, riding with my friends, many of whom are serious cyclers, makes me a better rider. As the ride progressed, I rode smoother, tighter and faster. Working on a Scrum team has a similar effect for team members. Knowledge readily spreads to the teammates through frequent communication via daily stand-ups and backlog grooming activities, making them stronger, better engineers.

Due to the fact that the ride occurs mid-Spring, each member of the team arrives with a different level of fitness, depending on how much training we put in on indoor trainers during the long New England winter. As such those of us less capable rely on help from the stronger riders. With cycling, teamwork is core to the effort. Riders work together, take turns leading the group, breaking the wind for the rest of the team, pulling the group, allowing others (slower, tired riders!) to draft behind them. Similarly, Scrum is focused on teamwork and less about solo achievements. Sprints (and thus product releases) can only efficiently succeed if the team works together and arrives at the end together.

There was a strong team commitment to finish the full 100K. During the ride there were several opportunities during the ride to cut it short - to quit early. Additionally, the event offers 2 shorter rides, of 10 and 30 miles. But we, as a team, committed to completing the 100K ride regardless of the weather or any other adversity that came our way. Scrum teams develop a similar strong focus and commitment to delivering potentially shippable product increments and overcoming impediments.

My friends and I had great weather and a fun ride. We reached the 100K mark, the “finish line” together with a team velocity of 17 [mph] story points. I’m looking forward to next year’s ride.

"Scrum is Like Chess"

2011-05-11T05:42:00.000-07:00

One of my favorite Scrum quotes is “Scrum is like chess.”

Like chess, the “rules” for Scrum are fairly straight forward. But also like chess, Scrum practitioners require continued practice and ongoing learning to be come good.

So why do it? If it is hard to master, why change our development process?

Ken Schwaber, a co-author of the Scrum Guide, sums it up best: “Those who play both games and keep practicing may become very good at playing the games. In the case of chess, they may become Grand Masters. In the case of Scrum, they may become outstanding development organizations, cherished by their customers, loved by their users, and feared by their competitors.”

Being loved by users and feared by competitors would mean that we’re delivering highly desired software products which are generating significant sales growth and, ultimately, more corporate value. This is the state we, as a company, want to achieve, and it is one of the drivers for our transition from Waterfall to Agile.

So what can you do to become better at Scrum?

Being part of a Scrum team and new to Scrum, you are likely devoted near-full time on learning new ways of working as well as actually delivering potentially shippable product every Sprint. But this shouldn’t stop you from allocating some time to digging further into the nuances of Agile software development practices, communication and organizational behavior.

There’s a wealth of information, as well as tons of opinions from many of the great Agile industry experts, on two public Internet groups. The first, and older, is the Yahoo! Group, Scrum Development (http://groups.yahoo.com/group/scrumdevelopment/). The second, a Google Group called Agile Leaders (http://groups.google.com/group/agile-leaders?pli=1) was recently created and also features some lively discussions, often by the same cross-section of folks on the Scrum Development list. By just becoming an observer to these lists you can glean many useful tidbits of information on all aspects of Scrum as well as Agile software development. While these are great resources for every Scrum team member, aspiring ScrumMasters and Product Owners should take particular note of these two valuable resources.

There’s also numerous blogs and webinars that can give you insight to different facets of Scrum. I’d start there before buying too many books (Amazon lists over 300 books for “Scrum” and over 1300 for “Agile”). There are many formal and informal Agile and Scrum gatherings, such as Agile New England (http://www.agilenewengland.org) that regularly host events and meet ups.

And finally, as with any discipline, nothing beats practice coupled with an ongoing desire to learn.

Enjoy the journey!

Multi-tasking Mayhem

2011-04-27T05:53:00.000-07:00

One of Scrum’s rules pertains to the purpose of each Sprint, which is to deliver increments of potentially shippable functionality that adheres to a working definition of “done.”

Scrum Guide: http://www.scrum.org/scrumguideenglish/

A few weeks ago I was part of a small team creating an Agile training course for our Product Group. The one-week time-boxed effort was broken into 5 one-day Sprints - we were going to eat our own dogfood and create the course using the Scrum framework. For our second Sprint, day 2, I volunteered to implement a couple fairly involved stories from our backlog, each requiring me to create a dozen or so slides with associated training notes. A accomplished multitasker, I dove into the effort, working on both stories concurrently. Late afternoon I suddenly found myself in the final hour of the Sprint with both stories more than half complete, but neither story Done! A bit of panic set in as I contemplated my mistake: if I had worked on the deliverables sequentially, I’d have easily completed at least one Story, come Sprint review time. As it stood then, I might not have anything done. I felt momentarily paralyzed by my situation and realized that I’d have to put in extra effort to get the work done in time. I also realized I had a great topic for my next blog, so all was not lost!

The ability to multitask is often viewed in the workplace as a positive skill: the ability to juggle and deliver multiple competing priorities in an increasingly complex and dynamic business environment. However, numerous studies have shown that multitasking only gives the illusion of enhanced productivity, when in actual fact, working on more than one task at the same time often results in less actual productivity! According to Psychology Today “the mind slows down when it switches back and forth between tasks”. Each time you switch to a new task you must gain context on the new task while losing some context on the task you left behind. A Stanford study suggests that when you are switching between multiple tasks, you “couldn't help thinking about the task [you] weren't doing”, leading to a further drop in productivity.

My take-away from this is two-fold:

Teams (and individuals) should work on one story/task at a time and deliver on stories sequentially. The prioritized Product Backlog and Sprint Backlog should is the artifact that focuses the Team.

Management should respect the Team commitment (of course!), keep team members 100% dedicated to the Sprint and do whatever necessary to eliminate competing priorities. When competing work is discovered, the work should be added to the Product Backlog and prioritized by the Product Owner.

Focusing one one story and completing that work before moving on to the next, Scrum teams (as well as individuals) will deliver more work. Sometimes you have to go “slow” to go fast.

Velocity and Release Planning

2011-04-13T05:07:00.000-07:00

“An iterative, story-driven process makes it easy to fix a date but difficult to fix what will be included by a given date.”

-Mike Cohn, User Stories Applied

Predicting a release date is often a big challenge in software development, particularly Waterfall-type development projects. Actually, making the prediction is the easy part - predicting accurately is the challenge.

Assuming a team is in place, determining when a software project will be released is a function of time, features, and quality. Further, by fixing any one of those variables, you can can expect to, perhaps greatly, impact the other two.

As my organization transitions from Waterfall to Agile, we’re challenged with how to accurately perform Agile product release planning and reflect that back to the wider (non-development) organization.   With the “old way of doing things” there is a general comfort with holding a written plan, a project schedule, even if everyone knows that plan has a high probability of being wrong.

Waterfall Release Planning

In today’s hyper-competitive business world, a product release date is often defined by company management. This is true in both Agile and Waterfall development environments. In my experience, it’s been true for most, if not all, of the large waterfall development project that I have been a part of the past 25 years.

Once a release date has been set and communicated to the field, the development team heroically tries to meet that date. If you are a lucky engineer, your management team gives you some time to flesh out requirements, create a architecture and design documents, define tasks and a schedule, and ultimately allow your data to influence the release schedule. Even with that luxury, challenges abound to meet this “defined up front” schedule: as you implement more, you learn more, the work remaining becomes more defined, ultimately impacting the project release date - often it is later than originally predicted, rarely is it earlier.

When faced with more work remaining than time allowed, a hard delivery date often forces you to cut features to meet the schedule. The problem is that features may be partially implemented (waste) and often most of the remaining work is quality assurance testing. In those situations, the choice becomes tougher: slip the schedule or reduce testing (and thus quality). It turns into a Kobayashi Maru (“no win”, for those who aren’t Star Trek fans) situation. We’ve all been there, and we all bear the battle scars.

Release planning is different with Scrum

With a Scrum Team producing potentially shippable code with every Sprint iteration, we’ve essentially fixed one of the release planning variables: quality. In order for a product to be potentially shippable, it must have a level of customer-expected quality. This leaves the organization two remaining adjustable dials: time and features.

The amount of completed work (user stories “done”) a Scrum team can deliver in a given iteration is identified as the team’s Velocity. Velocity is a key metric for release planning - it quantifies the capacity of the team for a Sprint. Knowing a team’s velocity, coupled with a sized Product Backlog, the Product Owner can get an approximate idea of what features will be delivered by a specific time.   [You can reference numerous books and blogs for further details about story sizing and velocity, with Mike Cohn’s User Stories Applied, and Agile Estimating and Planning being two of the more highly regarded references.]

Time-based Release Planning

If you’re organization is going to mandate a release date, you must readily accept that you won’t initially know the full feature set that will be delivered on that date. But, by using the Scrum Team’s velocity, you can get an approximate idea of the feature set.

Knowing a team’s velocity, how much work a team can get “done” in a Sprint, will help predict how much of product backlog can be implemented by the time the mandated product release date arrives.   By using the team’s velocity, the number of sprints that can be completed before the release date can be computed (remember, Sprints are of fixed time). Applying that to the backlog can give you a very good idea of what features will make the release and what are on the “bubble”. Usually this is better than good enough.   Features that are on the delivery bubble, those that may or may not make the release, are much lower in priority than those that will be fully “done” prior to the release date. These backlog items shouldn’t be make or break features, and can be saved for a subsequent release.

When the surrounding corporate environment is not Agile and you need to publish a future product feature list to a wider audience, the majority of the the features on the Backlog that comfortably fit into the velocity capacity can be communicated outward, setting expectations and enabling the coordination of public release-type activities such as PR, marketing, etc.

Feature-based Release Planning

If you product must complete a specific set of features before you can release, then using team Velocity can help you predict your release date. By using the team’s velocity, and mapping it to the product backlog, you can predict many sprints it will take to complete the required backlog items.   Release burndown charts are great at visualizing how much work is remaining in a release and identifying how many sprints are likely required before you can ship the product.

Because the team velocity can vary from sprint to sprint a firmed up average, team velocity usually isn’t firmed up until the team achieves a regular Sprint cadence - usually 3 or so sprints. To determine a release date early on, you will need to consider velocity variability. Mapping out a release date window can be computed by using the predicted best velocity and predicted worse velocity, which will give you a release window as your target release date. As more sprints are delivered, this release window can be refined.

Product Backlog Considerations

The Product Backlog is a living entity. As you learn more about your customer needs, and hopefully get customer feedback on early sprints, you likely will add and remove items as well as re-prioritize items, all of which can effect your release date. Additionally, the team velocity likely will fluctuate - but with short (2-4 week) sprints, you will have your finger on the pulse, having up-to-date velocity measures as well as average and historical values. Using this data, you can frequently map out how many sprints will be required to deliver true product value to the market.

Continuous Planning

Release planning within Scrum doesn’t happen just once, at the beginning of the release, as it does with waterfall development. The release plan is evaluated and possibly adjusted at the end of every iteration, as more is known and team velocity is refined. Realize that it may not be the project release date that is adjusted, rather it could be the work that will be delivered. As more potentially shippable product increments are delivered by the team, the organization has the opportunity to gather customer feedback, and use that feedback to influence (to add or remove items from) the product backlog. Doing so may change your release date, or the features ultimately delivered. Further, performing this activity regularly, doing frequent customer validation and backlog/release grooming, gives the organization the opportunity to deliver a product that better matches market needs.

For my organization, Scrum has given us a higher level of confidence with our release planning. This year we’re part of a new corporate initiative requires that requires product by the end of 2011. After the recent success of our Scrum pilots, we have a new found confidence in our planning such that we believe we’ll be able to deliver working shippable product by that deadline by using Scrum. We can’t currently say what the complete functionality set will be with absolute certainty, however, but that’s okay. We know we’ll deliver shippable product by the end of the year.

Transparency: Keep it visible!

2011-03-31T05:38:00.000-07:00

“Openness is required for Scrum to work.”

-Ken Schwaber, Agile Software Development with Scrum

Openness, transparency, visibility - no matter what synonym you use, it is critical to Scrum. My favorite term is transparency: Scrum’s openness makes everything that is going on within the project transparent to the world.   With Scrum there are no hidden priorities, no hidden work, everyone should (and generally does) know what is currently being worked on and what will be delivered by the team.

On the surface, it may seem that transparency is more important for management and the business owners rather than the team. Management primarily cares about when we can actually ship and start selling a product (and of course, that the product does what it needs to do - that it meets market expectations). Numerous business-related activities such as marketing, support, training, etc, are coordinated around a product release.

But transparency is also extremely important for the team and individuals. Nowhere is this more apparent than in a Waterfall development environment. There have been several instances in my career where a peer has asked me for workplace advice. The conversation zeros in on a boss who is giving them grief about their productivity. After listening to them vent, a common thread usually appears: these individuals often are doing a lot of work, usually juggling many different activities, but their manager is only tangentially aware of the scope of responsibilities this person is shouldering. The manager is not aware of their employee’s “backlog”.

My advice in these situations is simple: make a list of work you are doing, order it according to priority, and discuss the list with your manager. In particular, I suggest that they work with their manager to help set the right priority of the various work items. This conversation often opens the eyes of both the manager and the employee, quickly addressing the tension and frustration experienced by both parties.

With Scrum, this communication activity is commonplace. In fact it is core to the Scrum framework: the prioritized product backlog and the Sprint backlog are visible to everyone. As a result, the work being done by the team is obvious and very transparent. Further, project progress is demonstrated to the business owners regularly during the Sprint Review, via a (potentially shippable!) working code demonstration.

The Inevitable New Unplanned “Top Priority” Work

In a Waterfall environment, the impact of new, mid-project “high priority” work is often not immediately felt. Because the product release can be many months away, incoming hot issues can be dealt with in a timely manner and everyone feels good, at least in the short-term. However, this new work causes the team to accrue a form of “schedule debt”. The painful balance comes due at some point in the future, usually in the form of a schedule slip. And often, sadly, that schedule slip is a surprise to management.

In a Scrum environment, transparency is the proverbial Big Stick when unplanned “critical” work is added to the project. When management (let’s face it, it usually is management) adds new new work to the team’s plate, the impact to the team is immediately obvious: 1) the new work will either displace existing Sprint priorities causing the Sprint to be abnormally terminated, something everyone wants to avoid, or 2) the work will be added to the Product Backlog and prioritized according to business value. There’s no magic, no silver bullet, and everyone Scrum-enabled, will should understand that!

Surprise!

In software development, we don’t like surprises - we like predicable. Usually a surprise means new unplanned work for the team or a schedule slip. Someone is is going to be unhappy!

Scrum’s transparency helps organizations eliminate surprises or at a minimum deal with them effectively.   Surprises in the form of schedule slips are eliminated because what the team is working on is very visible and demonstrated at the end of every Sprint. Work, delivered via Sprints, is time-boxed and ends up being very predictable. Surprises in the form of new, unplanned work is dealt with effectively by adding it to the prioritized Product Backlog. With a prioritizing Backlog, the team is always working on top business value items.

Transparency, via a published, readily viewable backlogs allows everyone to understand what is currently being worked on as well as the impact of new unplanned work.

“Scrum keeps everything about a project visible to everyone.”

-Ken Schwaber, Agile Software Development with Scrum

Ship it! - Scrum’s “Potentially Shippable” Product Increment

2011-03-17T08:03:00.000-07:00

One of the problems many teams struggle with when beginning to adopt Scrum is delivering potentially shippable increments of product at the end of every Sprint. Delivering a potentially shippable increment of code is one of the fundamental concepts of Scrum - a requirement that is specifically called out in the Scrum Guide:

“Scrum requires Teams to build an increment of product functionality every Sprint. This increment must be potentially shippable...the increment must be a complete slice of the product. It must be “done.” Each increment should be additive to all prior increments and thoroughly tested, ensuring that all increments work together.”

Scrum Guide: http://www.scrum.org/scrumguideenglish/

Producing a potentially shippable product in 2 or 4 weeks involves a semi-radical mindset change to those used to building software in a traditional Waterfall-like manner. Making this mindset shift enables an organization to experience numerous benefits and efficiencies, from the ability to deliver software to customers at more frequent intervals, to being able to strategically pivot product direction while minimizing waste.

At first, most teams new to Scrum struggle with delivering software in this manner - our Agile Pilot teams were no different. The first challenge was integrating cross-functional team members, development, doc, QA, into a new team. Agreeing on the definition of “Done” helped with this. Another hurdle involved Product Backlog user stories - often user stories were too large, or difficult to break down into small single feature product slices. Backlog grooming quickly rose in importance for most teams. Team member specialization also presented challenges. Tensions around creating a shippable product often developes when team members such as QA and documentation have to complete there work the last day or two during the Sprint. At present this is our biggest challenge - the teams are still learning and adapting, as is the organization as a whole.

Over the course of a handful of sprints the Pilot teams put in a valiant effort, overcoming many of these challenges. Teams adjusted the way they worked, via Sprint Retrospectives, to address some of these concerns, and ended up producing high quality software and documentation. One of the Pilot teams even released the results of one of their early Sprints to customers via a Technical Preview (TP) pre-release program.

Last week I was fortunate enough to attend an executive briefing scheduled with one of our customers who has been participating in this Technical Preview program. As part of this TP, we supplied a documented script that the customers should follow; this script defined the usage pattern we used as a final test before releasing the product to this program. The customer, being a self-proclaimed power user, told us that he followed the script, but really wanted to see what else the product offered. He strayed off script and began investigating how the tool’s functionality could help his applications. I think a few jaws dropped in the meeting, as some were expecting the worse - straying off the prescribed path was an “all bets are off” scenario for pre-released software in our Waterfall development environment. Surprisingly to most, the unexpected happened: The customer told us that the software had all the functionality he expected and had very few bugs. Further, he stated that he was able to use the product to map how he would make use of it in his enterprise. He did mentioned, in a proud tone, that he found 2 or 3 bugs that he wanted to demonstrate to us. Sure enough, he brought up the software and proceeded to demonstrate how he used the software, along the way, attempting to show two issues he found. His subsequent focus on the issues, one minor (a browser-related issue), the other an issue he was unable to reproduce (but did note there was an easy work-around), didn’t minimized his earlier claim, now stuck in my mind: “the product had all the functionality [he] needed”!

It’s always great to see a Customer demo your software to you. Our Product Owner had an opportunity to gather real-world product feedback from a power-user customer, one who personally demonstrated how he would use our pre-pre-Alpha, but potentially shippable, version of our product. This customer’s feedback subsequently influenced our product backlog for an upcoming sprint - agility in action!

During this whole “customer demonstrating our product to us” segment of the meeting, I was smiling proudly: this was software that we, in comparison to a Waterfall-type release, had essentially “tossed out there”: no code freezes, no bug triage, no end-game QA bug fix cycle. It was product that was the Sprint 4 output from our new Pilot Scrum Team, software that the Team had deemed “potentially shippable”. And it was just that: potentially shippable.

Agile Pilot Teams - A Retrospective

2011-03-03T05:23:00.000-08:00

Early in our journey from Waterfall development to Agile, we identify 4 pilot projects, each from different product lines, to test the Agile waters via “Pilot Projects”.   Even though there were several Agile champions (myself included) in the organization, we decided to take a “let’s try it before we buy it” approach to Agile. Each of the pilot teams consisted of cross-functional team members. Our strategy was to formally train members of the 4 pilot teams as Certified ScrumMasters, Certified Product Owners or Certified Scrum Developers.

As the overseeing Unified Development Process (UDP) Committee, we identified our in-house Agile champions as coaches for each team, but ultimately each team had to self-organize, inspect and adapt, and deliver on their commitments following the Scrum framework.

Once trained, we sent the teams off to build software for a minimum of three two-week long sprints.

When it was all over...
When all Pilot teams completed their the minimum 3 Sprints we conducted a multi-team retrospective. Those of us that were coaching the pilot teams had already observed many of the positives and negatives identified during this retrospective, some of which are recounted in this blog. However, it was very heartening to hear similar sentiments expressed nearly unanimously by all Pilot teams.

What Worked Well?
The teams enthusiastically offered up a long list of “what worked”, but the first two items mentioned are worth noting here. Improved communication was unanimously agreed upon by all teams as the top benefit they experienced during the pilot. [Ironically, this conversation occurred mere hours after I had written my “Hyper-Communication: Scrum’s Secret Sauce” blog! I was mentally smiling as they spoke.] The teams stated that the Scrum framework fostered communication, mentioning that the cross-functional teams, specifically the inclusion of QA engineers, was extremely beneficial. Also singled out was the Product Backlog Grooming - it was a key team communication activity valued by all.

It’s interesting to note that the Waterfall process used in our Product Group does not prohibited nor dissuade people from communicating with others when developing software. I believe it is simply that often there is no immediacy to engage in these types of discussions in a Waterfall environment. The Waterfall process does not foster this type of regular extended team communication. For example, there is often a buffer, perhaps a project leader or supervisor between Product Management and the developers writing the software. Further, with Waterfall, the “are we writing the correct software” check-point is often distant, in terms of time, from the requirements definition, occurring at Beta perhaps, where as Scrum’s are much more frequent, occurring at the end of every Sprint.

The second notable benefit brought up was the feeling of empowerment experienced by the participants in the pilot.   This one is interesting and caught me a bit by surprise as it is hard to visually identify. The teams actually appreciated the complete ownership, not only for the delivery of working product, but for user story refinement, quality, documentation, etc. Success (or failure) was almost completely within their, the Team’s, control, and they found this to be extremely valued.

In Waterfall software development, most teams are used to be being led. Often a project leader or project manager is identified as the chief point of contact. That person manages the schedule, works dependencies, reports status for the team, and often runs interference, allowing the team members work on their individual tasks with minimal interruption. Teams are typically not cross-functional and often the onus of communication is often on the individuals themselves.

Scrum’s emphasis on “The Team” was appreciated by all of our pilot teams. The team members valued the empowerment and responsibility - essentially the confidence the organization put in them. As management, of which I am a part, it seems we really did succeed in stepping back and letting the Pilot teams operate without traditional management meddling.

What Didn’t Work Well?
The Pilot teams also brought up a healthy list of things that didn’t work. Being a new process that most had just learned, coupled with introducing Agile in a Waterfall environment, a long list of issues was not unexpected.

The number one issue the teams noted was that most Product Owners did not spend enough time with the team.   For our Pilots, we identified Product Managers as Product Owners for each team. Because our Product Managers were few in numbers and had many other responsibilities most ended up being only part-time members of the teams. The teams felt this strongly that the Product Owners needed to participate more as a full-time team member, attending the daily Scrum as well as spending more time creating and refining the Product Backlog and working with the Team on backlog grooming.

This issue was a direct result of mapping a Waterfall role into a Scrum role. In Waterfall, a Product Manager can manage many projects, defining the requirements up front, then moving on to the next project’s requirements. Not so with Scrum, where the Product Owner is a full-time job, a fact that became obvious very quickly for most Pilot teams. It’s clear going forward that our organization needs to re-examine our staffing and role needs, as well as acknowledge that some existing roles won’t readily map into new roles within Agile.

One other notable “didn’t work well” issue was the handling of customer escalations. With a large customer base, an escalation can come in and derail the team’s Sprint commitment. This situation can and does occur during our Waterfall-run projects as well. It’s an issue that likely will need an organizational solution, as it introduces tension between specialized engineers developing new software in time-boxed sprints and the need to support a significant and demanding and customer base.

We thought it was all over, but it wasn’t...
Much to my delight, all four of our Pilot Teams decided on their own to continue with Scrum beyond their initial 3-sprint Pilot. One Team is now on Sprint 7, and has successfully released the output of an earlier Sprint to customers in a Technology Preview program.

Beyond the team enthusiasm and product successes, the biggest outcome of our Pilot effort was that we created new Agile champions. Each Pilot Team has several members who have been positively promoting their Pilot Team experience. As we roll out Agile to more teams, we’ll leveraging these new Agile advocates as mentors and champions to assist with our organization Agile Transformation. To me, that was the biggest “what worked well?” of all.

Hyper-Communication: Scrum’s Secret Sauce

2011-02-16T05:00:00.000-08:00

Com com communicate

Communicate communicate

Via satellite and solid state

Never never hesitate

Communicate communicate

Never never hesitate

-Pete Townshend, Communication, from the album All The Best Cowboys Have Chinese Eyes

Effective and continuous communication is the cornerstone of any high-performance team. It may be obvious, but we need good communication to do our job well. But good communication doesn’t come for free: you have to continuously work at it. Further, when developing software, your choice of development process can hamper or promote high levels of communication.

During our Waterfall to Agile Pilot project effort, I’ve observed that Scrum has “forced” us to dramatically communicate more, and at more more regular intervals, than we have done in the past. Scrum forced our teams to develop a communication habit - one where where I would estimate that we communicate twice as much, if not more, than we did with our current Waterfall process.

Why is this so? It’s because Scrum prescribes a set of frequently repeating ceremonies (meetings) that require effective team-wide communication. Teams that adhere to Scrum develop a work cadence, a habit of working, that fosters regular, focused communication on the work at hand. Scrum promotes this team habit through a the following ceremonies:

Release Planning Meeting - held at the start of a project.

Sprint Planning Meeting - held prior to the start of a Sprint.

Daily Scrum (stand-up) - held every day of the Sprint.

Sprint Review - held at the end of every Sprint to demonstrate working software to the stakeholders.

Sprint Retrospective - held after every Sprint to inspect and adapt.

Several additional conversations generally occur regularly as part of developing software guided by Scrum. One, the subject of an earlier blog, surrounds the team agreement on what “Done” means. Another equally important conversation occurs with the Product Owner about the Product Backlog. Called Backlog Grooming, the team works closely with the Product Owner to focus on, understand, refine, and size top priority user stories over the course of every Sprint, a “just in time” process that helps feed the next Sprint Planning Meeting. The Scrum framework doesn’t require these conversations, but my observation is that they are regular ceremonies for most Scrum teams, and they were with our Scrum Pilot teams.

Contrasted Communication with Waterfall

Compared to Scrum, Waterfall-based software development seems lazy with regards to communication. A majority of team communication happen up front, when the least is known about the project. Up-front requirements, analysis, and design discussions help formulate a project plan and project schedule, then the team goes off and develops according to that plan. Regular project status meetings, usually weekly, are held to track the project and resolve dependencies. Sometimes, in fact sometimes quite frequently in my experience, status meetings are cancelled because everyone is “heads down” working, an ironic often-used phrase - If someone is head’s down, an image comes to mind of a someone cloistered, cranking out code, only coming up for air, for a bathroom break.

During past Waterfall development efforts, I have observed situations where an engineer, working “heads down” delivering according to his own schedule, is suddenly informed that a feature he previously completed needs to be re-implemented in a substantially different way, thereby putting him unexpectedly behind schedule. As a Project Leader I recall feeling surprised that this happened, after so much time being spent up-front identifying dependencies and coordinating task schedules. Something slipped through the cracks, causing the delivered functionality to based on stale, incorrect information. We simply didn’t find out that we went down the wrong path until much later in the release, a point in the project where it was more costly to fix the issue. These were situations that could have been easily avoided with just a little bit more communication with the appropriate parties, a situation much less common with Scrum.

Continuous Communication with Scrum

During two week Sprints, the Scrum Team is meeting and communicating at least once daily, with a 10% of the Sprint spent on planning, demonstrating product, and adapting and improving their process. Further, there are daily Scrums, discussions with subject-matter expert, and design discussions and activities between team members. In other words, plenty of opportunities where team members are in essence, forced to communicate.

One benefit I observed during our Pilot Scrum Team effort was the impact that the Daily Scrum had on team communication. The daily Scrum meeting frequently spawned ad hoc discussions immediately following the 15 minute stand-up. Team members responding to the daily Scrum “3 questions” fostered additional, unplanned, clarifying discussions, just by team members actively listening to teammates. Project questions and issues were often resolved immediately, or “just in time” rather than laying dormant until some point in the future when they would surprisingly be discovered.

To those used to Waterfall, the communication required by Scrum may seem like hyper-communication, perhaps overkill. But ultimately it works well. Teams quickly develop a communication rhythm, involving Stakeholders (via Sprint Reviews), Product Owners (via the Product Backlog Grooming) and fellow team members (via the Daily Scrum). With the Team focused on the same goal, delivering on Sprint commitments, continuous communication is the secret sauce that allows the Team to succeed. It’s the catalyst to turning a team into a high performance team.

Pete Townshend said it best:

“Communicate communicate, Never never hesitate”

Management’s Role in Scrum

2011-01-31T05:23:00.000-08:00

In one of my earlier Blog postings, I responded to a comment and stated that I believed that “Scrum aims to make managing software development easy”. I believe this is true because the core of Scrum is the self-empowered, self-organizing Team. In its simplest form, a Scrum Team is fed a prioritized list of user stories and the Team commits to delivering some set of top stories in a fixed amount of time. Because the team is empowered, it is up to the team itself to figure out to divide the work between themselves and deliver “done” stories by way of a potentially shippable product at the end of the sprint.

As an Engineering Manager, I can sit back and watch an established Scrum team crank through their work. I can readily track the Team’s work via burn down charts and the Scrum task board. In fact, I was able to do just that during our recent Pilot with the Tool team.

A fully-functioning Scrum team will ask for help from the appropriate people, usually Subject Matter Experts (SME) and Product Owners (PO), and will inspect and adapt to make their work more efficient. When any blocking issues occur, often called impediments, the team will raise these issue to their ScrumMaster. Nowhere in the Agile Manifesto or Scrum Guide is a role for Engineering Manager defined.

Because there is no need for day to day management and oversight of a Scrum team, the traditional software engineering manager may feel marginalized, perhaps assuming that their primary responsibility is writing annual performance reviews. Ugh, what a horrible role that would be. Thankfully this couldn’t be further from the truth, particularly during the process of transforming from a Waterfall process to an Agile one.

A Manager’s Top Priority

A manager’s top priority is to put their employees in a position such that they have the best opportunity to succeed. In a Waterfall environment, this usually means breaking down and assigning the work, working to identify and resolve cross-team dependencies, managing the team’s schedule, and communicating release status to upper management. The common thread for these activities is that they facilitate the Waterfall development process.

In an Agile environment, a manager’s goal is the same, to create the best environment possible for that allows their employees to succeed. But facilitating the development process in Scrum is very different. Initially with Scrum, a manager has to be a champion for the process and help employees unlearn roles and responsibilities they’ve grown accustom to in Waterfall. With Scrum, first and foremost, it is a Team effort versus an individual one. The Team makes a commitment to work together and do all the work necessary to deliver “done” user-stories. This can’t be understated. It may be hard for a manager to defer to the team and let the team self-organize, but once a manager “let’s go of the reins” and lets the team “figure it out”, their real work begins.

An Agile Manager’s Real Work

The first job of a manager is to assemble the right people for each Scrum team. The skills and personalities needed for a waterfall-oriented team are often drastically different than needed for an Agile team. Because Scrum teams deliver vertical increments of functionality teams members may need to be more cross-functional than siloed.   For example, rather than a team of User Interface engineers, a more appropriate team would include UI, middleware and database engineers, covering features from the front-end all the way to back-end.   This may require a more matrix-oriented organizational reporting structure or changes to the existing organization, something that may ruffle territorial feathers. But such changes may ultimately what is best for the organization, and a manager should accept and adapt.

Managers have a broader understanding of the business, and can also act as subject matter experts (SME), providing coaching to the team as requested, and additionally participating in the Sprint Reviews where feedback on delivered functionality can be voiced.

In addition, managers should also consider and account for external influences that may impact the team. Because the team has committed to a delivering features in a time-boxed iteration, managers should make it one of their top goals to keep the team from being interrupted. Team priorities are managed via the Product Backlog and short sprints give Product Owners frequent opportunity to re-prioritize work. Managers should strive to block interrupts during a Sprint, by either pushing back on the intrusion or pre-identify organizational capacity for predicted interrupts such as software maintenance. I could go on and on about this as I feel strongly about this point. The Team makes a strong commitment to deliver on the stories. Managers should respect that and equally make a strong commitment to protect them from interrupts and priority-shifts during the Sprint.

Perhaps one of the most important roles a manager has in Scrum is to address team impediments. A manager should regularly connect with the ScrumMaster to understand and work to eliminate impediments identified by the team. Impediments may be easy to remove, such as finding budget money for new hardware (perhaps not so easy!), or may sound impossible, as in fixing the timezone issues between the U.S. and India. Regardless, addressing impediments of all shapes is critical to the health and productivity of the team and it is ultimately managements responsibility to address these issues that can’t be addressed by the team. Impediments offer an opportunity to think out of the box and to operationally improve organizational efficiency.   Here, out of the box thinking and sometimes creative organizational changes can make a huge impact to the team. It’s a perfect opportunity for the manager to challenge “the way things have always been done.”   Removing impediments could almost be a full-time job, particularly for an organization transforming from Waterfall to Agile.

Some final thoughts on a manager’s role in Scrum: A manager should never be the ScrumMaster for teams containing direct reports, as this, in practice, breaks team self-organization. Some companies have, however, had success with managers acting as co-Product Owner. Engineering Managers partner with Product Managers, with the Engineering Managers focusing inward on the team, and the Product Managers facing outward to the customers. This role may work where the manager has a strong technical background and the team is without an Architect or strong full-time Product Manager.

Agents of Transformation

Over the past year we’ve asked each Agile pilot team to drastically change the way they work, by holding daily stand-ups, tracking and publishing Sprint burndown and regularly delivering potentially shippable software every 2 weeks. But as managers we’ve essentially kept our day-to-day work environment unchanged.

Converting from Waterfall to Scrum forces engineering managers to change their day-to-day activities within the organization. Rather than “owning” the project schedule, their role shifts to enabling and coaching self-organizing teams as well as removing impediments.   But I’d offer up that the role goes a step further: as Managers, the organizational transformation starts with us, the business owners.

Agile development challenges us to see beyond the “this is the way we’ve always done it” mantra. Just because it “is”, that doesn’t mean it “has to be”. And there lies the opportunity for transformation. As the management leaders of this company, it is our responsibility to not only facilitate this transformation, but to also partake in it ourselves.

The Burn Down Chart: A Picture is Worth a Thousand Words

2011-01-21T07:19:00.000-08:00

Back in the halcyon days of Digital Equipment Corporation (DEC) my team was “volunteered” to participate in a pilot process improvement effort called Software Metrics In Action (SMIA). As project leader of 10 engineers working on an 18 month project, we were a perfect candidate for this effort of inspecting and adapting and ultimately improving our software development methodology via Software Metrics. My participation in this process gave me valuable experience with process improvement. What I didn’t realize at the time was that I was being exposed to many of the key elements of the yet-to-be named Agile software development philosophy.

As I look back to that period of time, I recall measuring every piece of the software development process we could identify. Our team motto was “Data tells the truth about perception.” Seriously! We measured everything. We compared task effort ‘actuals’ with ‘predicted’. We tracked our time for development, test, vacation, dentist and doctor visits, even bathroom breaks, and we had graphs for every data series. But in the midst of this data collection and evaluation there was one metric that stood out. It was the one ultimate metric that the business owners, our management, was most focused on: the “work remaining”, specifically: when can we ship?

To track the answer to this question we kept a Microsoft Project schedule that contained only the remaining work.   Each day we adjusted the start date of the project each day to the current day and recomputed the remaining schedule. As more project work got completed, our knowledge grew and so did our confidence in our task estimates and our release date.   By continually refining our estimates based on new knowledge, we were able to track, in “real time”, the effective project completion date on a daily basis.

My teams effort in our1990’s pilot organically stumbled upon one of the key mechanisms Scrum uses today to track project status. Scrum monitors “work remaining” by using the concept of Burn Down, and in particular, Burn Down Charts.

Burn Down Charts

There are two types of burn downs that are generally tracked by a Sprint team, the Sprint Burn Down and the Release Burn Down. They both operate similarly, but have a different scope. The Release Burn Down most closely matched my DEC pilot schedule release date monitoring, as it tracks work remaining for the entire project. The Sprint Burn Down tracks work remaining within a single Sprint, usually 2-4 weeks long. Because the Progress Agile pilots were of such short duration, each Progress pilot focused solely on Sprint Burn Down Charts.

Understanding how to interpret these charts is important. As a business owner they provide a valuable and near-instant view into the health (or status), of the current effort, be it a Sprint or Release.

Reading the Charts

A burn down chart is laid out simply: The x-axis denotes number of days in the iteration (or project), the y-axis, for a Sprint Burn, is amount of available time in hours (for a release burn down, it is story points - topic for another blog). The numbers on this axis are computed by multiplying 8 (hours/day) times the number of team members, assuming each team member is full-time on the effort.
Reading a burn down chart is fairly simple once you understand what to look for.   Burn down charts show a the rate of progress of work being completed compared to the ideal rate of completion of that work. All burn down charts have one thing in common: the ideal burn down line with the ideal slope. Note the pink line in the Sprint burn down chart taken from one of our Agile pilot teams. This pink line shows the ideal rate of progress, specifically that one day of work (8 hours times the number of team members) is completed each day. There are 5 days of work, the Sprint iteration is 5 days long, and at the end of the sprint, 5 days of team work should be completed.

The pink line is boring. It’s boring because it is our control value, it’s slope is always the same. In this chart, the interesting line is the blue line. The blue line denotes the predicted remaining work for the iteration, in essence the current remaining workload of the Sprint. If everything works out as we hope, at the end of the sprint, day 5 in this example, the blue line will converge exactly with the pink line, at zero work remaining (since there is zero time remaining!).    If we examine this chart on day 2 and day 3, we can see that the team is ahead of schedule, because it is below our control line. Possibly the team under-committed to work, but regardless, at day 2 it looks like they are on track to meet their commitment to complete everything they said they would.   By day 4 the team is exactly on track. But something happens by day 5 because they are half-day behind. Perhaps a team member became sick or perhaps the team discovered some major unknown work at the last day. Either way, this is likely a bad example I have chosen to discuss because the team missed their commitment and everyone only found out about it the last day. The retrospective for this Sprint must surely have been interesting! This chart introduces more questions than it answers, but it encapsulates a interesting story. It turns out that this sprint was a bug fix sprint, in essence a “release sprint”, designed to produce a customer-shippable technical preview kit. Bug fixing uncovered further bugs and the team was unable to fix everything they found in time for the release, the end of the sprint.

Yes, that burn down chart tells quite a story, a thousand words, as they say, and if you are the business owner, it’s your quickest and best view into each project team.

More Examples
Let’s look at a couple more burn down charts. In this chart the team is tracking close to the ideal progress line, the black line in this chart format.

The remaining work, Task To Do blue bars, are flirting above and below the ideal. With 2 days remaining, it is likely this team complete all of their committed work. In fact, because the remaining work amount hasn’t changed the last few days, it suggests that an inquiry to the ScrumMaster might be needed to see if people simply haven’t been updating work remaining in their effort to complete the Sprint.

In this final chart below, it looks like the team has over-committed, almost from the start, and true to form, they were unable to complete a bunch of work, just over two days of work.
As you become more comfortable with read burning down charts, you will be able to make project status assessments in a matter of seconds. Tools such as Rally or VersionOne automatically generate these charts based on user data, or, as with one of our Agile pilots, a standard spreadsheet can be created to plot the team data.

Scrum Metrics

Metrics capture what happened in the past. With historical metrics, we hope that we can understand the past and adapt more efficiently to similar conditions in the future. While Scrum values inspecting and adapting, Scrum looks forward for iteration and release status using burn down charts.

Incidentally, my Digital Equipment Corp Software Metrics in Action pilot was deemed a success: our 18 months development project was released 1 day ahead of our original schedule. You can’t change the past, but sometimes you can predict the future...

What does “Done” mean?

2011-01-12T07:15:00.000-08:00

It starts with a team conversation...

One of my favorite team conversations that generated by the Scrum framework is one that occurs at the beginning of a Scrum project. It’s the team discussion and subsequent agreement of what the word “Done” means.

A Scrum Team is supposed to produce potentially shippable code at the end of every Sprint. This means that any stories (requirements) that the team commits to must be full implemented, documented, tested, etc within the time-boxed Sprint. The stories must be DONE. Done can and often is defined differently by each team. Minimally, Done means that the story committed to is coded and fully tested. A story really can’t be potentially shippable if it isn’t coded and tested.

Done is demonstrated during the Sprint Review, where working code is demonstrated the business owners and everyone else in attendance.   A story isn’t complete unless it satisfies the definition of “done”. In fact, Scrum will not credit the team with story points unless a story is, in fact, “done” (and accepted by the Product Owner).

Each Pilot team that I observed went through similar discussions about “done”. Because Scrum was new to the teams, the initial definition of “Done” was a bit loose: there was general agreement that a story was “Done” if it was coded and tested.    Over the course of the first Sprint Planning and the start of the Sprint 1, it often became apparent how loose that definition of Done actually was.

The Product Owner and Team conversation for story acceptance criteria was one of the main catalysts for Done refinement.   As each member of the cross-functional team discussed the stories they were committing to, it was quickly realized that the initial definition of done left many questions unanswered. Product areas such as documentation (what kind? what level?), training (again, what kind?), automated tests (how much should be automated? what platforms should be tested? what web browser? what about performance testing?) and bugs (should all be fixed? just blockers? more than just blockers?) all had questions that needed answers.

Over the period of a month or so, the teams each refined the meaning of Done. The process of discussing, debating, and ultimately answering these questions helped the group grow into a high performance team. This was very important to the success of the effort, as with Scrum the commitment to delivering a story is a team commitment, rather than an individual one.

A Great Habit

I think most would agree that it’s great to get in the habit of fully completing work, versus completing 80% of the work and leaving the remaining 20% for some time “later”, during the release end-game, or perhaps never (thereby wasting effort). With Scrum, producing done work is essential to the process. In the pilots, I’ve noticed a tendency for teams to commit to too many story points during a Sprint, ultimately leaving one or two stories partially finished, though the team put in often heroic effort to meet the original commitment. The unfinished stories were fortunately completed in the next sprint, but the fact that they were not complete often defeated the “potential” from the potentially shippable product.   One way to deal with this is for the team to sequentially complete stories within the sprint, never starting a new story until the current story is Done. This isn’t always practical, or efficient, but the team should be aware of the story bleed and velocity drag that partially implemented stories create. Alternately, it is perhaps better to commit to fewer story points per sprint and fill in any capacity with product maintenance or backlog grooming or architectural spikes.    These are team decisions that can be considered during Sprint Planning or during the Sprint Retrospective.

Agility

For me, the concept of Done is the key foundational principal that makes Scrum agile. At the end of a Sprint all stories are fully completed, thus potentially shippable. This fact, coupled with a short development iteration, offers the business frequent predictable time windows in which to re-prioritize work or introducing completely new requirements. Scrum allows an organization frequent opportunities to, in contemporary terms, pivot. Easily and readily shifting priorities with minimal impact to productivity is one of the primary mechanisms through which an engineering organization gains agility.

Scrum @ Progress

2011-01-03T10:16:00.000-08:00

Earlier in 2010, my company began the process of defining a Unified Development Process for the 500+ engineers in the Product Group. We formed a committee containing key representatives from across the product group. After extensive investigation and deliberation, the committee chose to focus on an Agile process, and in particular, Scrum.

What is Scrum and how does it fit within our Company?

Scrum is a software development framework, and not a prescribed process. It’s a framework that focuses on cross-functional, self-organizing teams producing potentially shippable business value in increments called Sprints.   A quick Google search will uncover reams of literature on what Scrum is, with the Scrum Alliance has a nice short overview here: http://www.scrumalliance.org/learn_about_scrum.

What’s important about this process effort is how such an Agile framework fits within our existing 25+ year old corporate environment. On the surface it sounds simple: educate the engineering teams, flip a switch or shout”on your mark, get set, go!” and voila!, we’re now Agile! In reality, evolving from a Waterfall software development lifecycle to Agile is a monumental undertaking, but one that is, in my opinion and experience, worth taking.

As part of the our effort, we’ve began running a number of Scrum Pilot Projects, projects in each division who have “volunteered” to build software using the Scrum framework. I’ve had the good fortune to “own” one of the pilots and to also observe others. Most of the benefits and challenges that I have observed during this process matched my own experience as well as those I have heard about from friends or read about from others in the software industry. And some of what I observed surprised me. I hope to capture these observations as the subject this blog, aptly titled “Agile making Progress”.

Scrum Roles

There are three roles that the Scrum framework identifies: ScrumMaster, Product Owner, Team Member. How the roles do or could fit into my company's Product Group is discussed below, ihowever it should be noted that every group, every product, has potentially different needs.

The ScrumMaster. The ScrumMaster is the person who ensures that the team is functional and productive. The ScrumMaster role can be viewed as the facilitator or ringmaster. At my company the role of a GEM (General Engineering Manager, a project coordinator) may align well with ScrumMaster. For the Pilot Tooling team, the pilot project that I own, we’ve got a great ScrumMaster who is in fact a GEM. However, this shouldn’t be a hard and fast role mapping. Other teams have ScrumMasters who are engineers, or even engineering managers (just not the manager of anyone on the team). The ScrumMaster cannot hold a position of reporting authority over any of the team members. On any team, it is very likely that the “right” person for the role will rise to the occasion - someone who is a facilitator or coordinator of the group. With the pilots, sometimes the choice was obvious, as it was with the Tooling team, sometimes after a sprint or two, the team realizes that someone else on the team might be better suited as the ScrumMaster and the team adjusts, as happened with another Pilot team.

The Product Owner. The Product Owner is responsible for the business value of the project, and represents the “voice of the customer.” The PO is also identified as the “single wring-able neck”, having ultimate responsibility for the project. On of the most important tasks for the Product Owner is for identifying and prioritizing the “stories” (release requirements) that the team will be working on. Here in in my company, one initial thought expressed was that a Product Manager should be a part of every team in the role of Product Owner. In theory this makes a lot of sense. In practice, however, it quickly becomes problematic. We’ve seen in the Pilots that the user stories and product backlog require near-constant refinement (also known as “grooming”). Additionally, some user stories are very technical in nature, due to the type of software we are building. A high degree of low-level technical understanding has often pulled in a Software Architect to help with backlog grooming and prioritization.   The Tooling team found that the backlog grooming and story refinement regularly required deeper technical expertise. Scrum favors empowered and self-organizing teams, and a this team quickly readjusted to solve this need by making the Product Owner a joint role, shared between a Product Manager (outward view) and a Software Architect (inward, technical view). To date that partnership has worked well over 4 sprints. It will be interesting to see how the Product Owner role will scale within the larger organization where there could be a dozen or so Scrum teams working on the same product line. Perhaps we’ll see some form of Product Council or co-Product Owner or a hybrid approach. I look forward to seeing how each team self-organizes!

The Team. The team consists of the people doing the work. As a general rule, the team should be cross-functional, because at the end of each Sprint a potentially shippable piece of software is produced. For software to be shippable it must be written, tested, documented, packaged, and have training materials created. As you can imagine, this generally means that you need Developers, QA, Doc and Readiness on the team. The Scrum team is given the authority and empowerment to self-organize and do what they need to do to get the job done. In return for this self-governing power, the business owner asks for a team commitment to deliver some level of potentially shippable software in a predetermined amount of time. In other words, the team commits to implementing a some number of stories (requirements prioritized by the Product Owner) within some predetermined amount of time (generally 2-4 weeks - for the pilots it was 2 weeks). Because I had a high level of confidence in Scrum to begin with, I wasn’t personally surprised by the (tremendous) output of the pilot teams. I was surprised, however, by a higher than expected commitment by each Team to make this effort work. This was Scrum magic, in my eyes.

Organizational Transformation

Due to corporate evolution, the environment I work in makes it hard for many us to focus on one project.   Because we’ve all got a number (often different) “top priority” work items, we must often time slice our attention across these efforts to achieve some level of success. This strategy is at odds with Scrum, as Scrum requires a Team to commit to deliver an unchanging list of functionality, over a fixed period of time. In other words, Scrum teams need unwavering focus on commitments they have made. For a Scrum team to succeed, Business Owners (management) need to leave the team alone during that time-boxed sprint so that the team can succeed. Here, to me, lies the biggest organizational challenge.

The pilot teams demonstrated to me that even in the face of major impediments and overriding “external” priorities that they were able to “figure it out” on their own and make it work. This is a good thing and gives me hope for the success of our ongoing organizational transformation.

Agile Transformation and Me

2010-12-28T09:53:00.000-08:00

I've worked for big companies like Digital Equipment Corporation, medium, like Sybase and Progress (my current job) all the way down to small start-ups, like EasyAsk and Novera Software. Each job has offered me different opportunity to grow, from create value from nothing at start-ups, to managing product lines generating hundreds of millions of dollars. Through it all, one thing has remained constant regardless as to the size of the company or my role in it: continuously improve and become more efficient. The only difference I've experienced is that with smaller companies it is generally easy, or quicker, to change what (and how) you are doing. The larger the company, the more difficult it is to change. Pure size, inertia and existing customers and revenue provide significant reasons for doing things the way they have always been done.

It is in this type of "bigger company" environment that the greatest challenge and perhaps the greatest rewards lie. Improvements in the efficiency of the software development process can reap significant dividends. Over the past decade+, a major evolution in software development practice, in the name of Agile Software Development, has been taking hold in the industry. I've been fortunate to have experience with delivering product using Agile frameworks and methodologies, both as an individual contributor, project leader, up to VP of Engineering. In my present role of Director of Engineering for 3 product lines generating over $300M in revenue, I am one of the leaders working to convert my the engineering organization of over 500 software engineers from a Waterfall-centric development process to Agile (Scrum) process. As this transformation got underway and I began coaching our pilot teams, I found there were many seemingly random thoughts and ideas bouncing around my head that I started writing them down, and ultimately started blogging within my organization. I've decided to post these blogs publicly, charting my organization's journey, our Progress, from Waterfall to Agile.

Thanks for reading...

John Piekos