Scrum Based Development

Agile. A newer perspective

2014-01-22T13:20:00.000-08:00

I think agile has traditionally been used in engineering environments for software development. You see it in hiring requirements for engineers, product managers and leadership. I think agile makes sense in the larger context - the business context. Imagine making your business model Agile and your business itself agile to disrupt industries?

Let's look at the disruptive industries with some examples that I think are interesting to look at:

1) Apple - they brought us itunes, ipod, iphones and the macs. They were failing a few years ago before Steve Jobs took over. They became agile in their business, to look for new industries to disrupt. They launched the iphone in June 2007, and have relaunched releases roughly every 1-2 years. It has become the largest revenue generator for apple. I am pretty confident, they have looked at their business and made it agile to be flexible to take over market share.

2) Sony,phillips,toshiba - the walkman, portable cassettes - displaced by the CD, displaced by the DVD and now replaced with an ipod and most recently with streaming itunes and spotify or pandora. The amount of time has compressed over the recent years for disruptive industries to make older technologies obsolete.

3) Tesla - electric vehicles - the model S launched in via a press conference in 2009 and started production in 2012. This is a brand new platform for electric vehicles, which if we think about it, is a conservative industry thats slow moving and created a new market in roughly 4-5 years. I think that's incredible given the type of industry they are in.

I think industries and companies that are not agile in their business will lose in the long term. Over time, the cycle of being disruptive will compress.

thoughts ?

Agile is easy on paper - how to address the intangibles

2012-04-09T12:50:00.000-07:00

I recently gave a talk at the rally agile cafe conference about Agile and SCRUM practices and I was really intrigued by how common the evolution of agile is across industries. There were several examples of Agile implementations with rally and we crossed industries from consumer electronics, to fortune 100 companies like John Deer ( where they use Agile to build robot controlled tractors with GPS systems) and enterprise software. What struck me was the same difficulties arise when you traverse the agile maturity curve - what I mean is the following

1) Culture - All the books talk agile in a fairly straightforward way - but what happens in reality is that culture comes into play about how you really want to absorb agile. The VP of Eng of the consumer electronics company was acknowledging that its not just good enough to put in the tools and practice in place. I feel that you really need to graduate to the "culture of agile". The "culture" of agile means, putting in the right practices around what it means to implement agile, how you communicate to your customers on releases, what the impact this has on your products. In order to do this right, the business has to transform in how it works so that the right expectations are managed. Unless one doesnt really think about that, then you dont really implement agile in the right way. It means, Agile has to be embraced at the exec level and fundamentally understood.

2) Ecosystem - Agile has to exist in an ecosystem of the right toolset so that you have the right level of visibility to other organizations and their ability to manage. Visibility to me means better predictability, staying one step ahead. Really respecting the statistics you get out of the toolset(I have used Rally and Scrumworks the past). For instance, velocity charts, and understanding the gradient of the work product and seeing whether you will really make it. The ability to figure out how many bugs you think you can fix in a release and making some hard choices on where to push back and which features are key. The other part of the Ecosystem is how the impact of particular feature has on customers and how your product owners evaluate risk of features and their scope. This impacts sales and possibly product services organizations that need to work with the product.

I firmly believe that a successful Agile shop takes into account all of these factors in order to provide a more predictable plan of attack and a better way to manage product and customer expectations.

thoughts ?

Scrum - play by rules

2011-05-22T17:41:00.000-07:00

A recent post by Ken Schwaber, on his blog made me think on the real pressures and culture of some of the workplaces I have encountered recently where that say they implement SCRUM and they claim their implementation fits their culture and company. Upon closer examination, there are short cuts, there are aspects where teams seem to adjust the process. To me it seems these teams are scared to go all in with the process, since the process of SCRUM is very well defined, the rules are known and its a question of getting better with time as you use the methodology. Perhaps they think they are better than SCRUM in some way by making these adjustments ? or perhaps just lazy?

I am not questioning that it takes time to absorb the concepts, make use of them, learn from your mistakes and get better. Everything gets better with practice and I don't believe SCRUM will fail you, its safe, if executed correctly.

Have a safe SCRUM - it won't fail you.

Importance of team structure and risk evaluation

2011-05-20T11:35:00.000-07:00

I was talking to a couple of software professionals recently and they employ the use of SCRUM and what I found interesting was that they all said, "we use N scrum teams" and "we have been using it for a while now ( over 1 year or more) " but it seemed to me that achieving the right level of communication was still missing and the deadlines were somewhat missed. I asked them , how do structure the team and how do you evaluate risk ? They all said, we have teams that are focused on particular competencies like network, or applications or platform ... they all use accurate stores to define the problem to be solved ( which is ok )... but they all had cross communication issues, which in effect caused a "silo" situation among the teams.

In my opinion, I think one really needs to look at the teams structure and ensure that the right representatives are part of the team, for instance, if you have a feature that requires operations, and you are deploying to a website, then its natural and a must, to have the operations folks be part of say the application team standup. I strongly believe that the team should be structured as a feature team which needs to include all participants impacted by that feature throughout the planning and execution phases. This increases transparency and understanding early on so that when it becomes time to deploy or ship, you are in a better place and less riskier place to succeed. The team structure and risk are really separate concepts, but I think merging them together makes sense and adds to the overall reduction in shipping risk.

In addition, as the stories are getting figured out, its important to understand the "feature risk" and the probability of impact and understanding the scrum risk curve.

Scrum risk curves, would involve understand the feature in a few sentences and its risk, the probability of success or failure, the exposure and time lost. This is a concept originally introduced in 2004, by John Brothers in The Agile Times in 2004: the risk burndown chart.

Figure 1: Risk Census

Our simple risk census here describes each risk, provides an estimate of how likely the risk is to occur, the impact to the schedule if the risk did occur, and then the expected exposure to the risk which is the probability multiplied by the size of the loss. I recommend creating a risk census during the first sprint planning meeting and then updating it quickly during subsequent planning meetings as new risks are identified or as the probabilities or sizes of known risks change.
The risk burndown chart is then created by plotting the sum of the risk exposure values from the census. My recommendation is to sum only the top ten risks even if the team has identified more. Do this even if the top ten change over the course of the project. A risk burndown chart will look just like a normal burndown, except that it plots risk exposure over iterations.

Definition of Done (DoD) and Quality - are they the same?

2011-04-14T11:44:00.000-07:00

The concept of Done and its definition is general can and will vary based on teams and teams definition , but overall, from what I have read and experienced in implementing SCRUM, you have the following concepts making of DoD or a variation of the items below.

Short Spec created
Coding/Unit Tests created
Acceptance Tests created
100% Acceptance tests passed
Product Owner demo passed
Known bugs fixed

I think that DoD is actually dynamic in nature and depends on business needs, schedule and risk, the definition of Done can change however there is a component of software that in my mind is a non-negotiable which is the concept of quality. So how can we balance the notion of quality, definition of Done and meet deadline delivery and promise of "potentially shippable software".

I think the answer lies in making the right tradeoffs in terms of risk and having the right strategy in place in understanding feature complexity, ability to break things down into small consumable features that allow the team to achieve the level of confidence which includes QA and development and having the key indicators to be there. I believe that pushing quality up the chain in the development cycle is key , by having continual integration in place, understanding the user story in detail, so that unit tests, QA plans and code reviews as well as bugs address quality fundamentally early on.

Just because you have a definition of Done agreed to, doesn't necessarily mean you will get the quality you need out of the feature set and release you are looking for.

thoughts and comments welcome, and my final contribution in the post is a great book about the pitfalls of development which I think many people already are aware of, which is a great read :-).

Code Complete: A Practical Handbook of Software Construction

Comments?

Work in Progress and Done

2011-03-30T12:01:00.000-07:00

One finds that when SCRUM teams start with development, the team takes on a bunch of tasks and starts marking them as Work in Progress ( WIP ), so you find yourself seeing a sharp spike upwards on the WIP status which results in an interesting situation as the burndown occurs.

You might think that this is correct practice, getting many things fired off at once and then converging down to "done done". Well, based on my experience and articles out there, that's not a good thing. Its far better to get to done items quicker in the sprint by having teams focus on getting the tasks done vs the "spray gun" approach. What can happen is that you can mark that 80% of work is done but only 20% of your features get done, there is a great example of this when google was working on the the adwords project ( I also experienced similar behaviors with teams when I first implemented SCRUM at my previous companies).

The best technique is to put a couple of engineers to work on the particular feature and tasks in a more serial fashion so that the features get "done done" and move to the next item in the list. You drive quality and reduce the "work in progress" gaps

Here is a chart ( borrowed from Ken's talk at google, that represents the idea). Too many tasks, marked in Yellow, represent tasks which are started but in complete, White represents the Burned tasks("done") and Red are the not started tasks. The goal is to make the burndown more lean and reduce the amount of parallel tasks which are started and increase the amount of burndown tasks which are "done". The bottom line is that the higher the white in the chart below the higher the quality and features complete and the better chance you have in finishing, versus, the higher the yellow in the chart the more risk of having a slower velocity in completion.

thoughts and comments are welcome.

Using Waterfall and SCRUM together?

2011-03-24T02:07:00.000-07:00

Can it possible to even think of the idea that Waterfall and SCRUM can me combined? Well, I was talking to a CTO of an established company, 350 employees, 70 million dollar run rate-- they have over 100 engineers and established product and they just moved to SCRUM about a year ago. It was an interesting conversation and I would love to get opinions on this as I have some of my own.

Here are some snippets from the conversation...
- Waterfall is a great way to communicate releases and dates
- Scrum can do the same but its a continuous process with no seeming end-date ( at least thats how some people perceive it)
- Execs are used to Waterfall where concepts of GA releases have a ship date

In a recent project at Oracle I was able to use the strengths of both approaches in helping deliver software. I am curious if there are others sharing similar or different thoughts on this. One could argue that if we needed both then perhaps SCRUM wasn't followed that strictly since one can group stories and backlog items into a release.

comments welcome..

Engineering operations and SCRUM

2011-03-23T23:34:00.000-07:00

So, now you have decided to take the plunge and start working with SCRUM. That's great...BUT, it isn't enough to get to you cross the finish line. There are some key engineering processes, in my mind, having lived through the horror stories of not having these in place, that need to compliment the SCRUM process which teams may or may not be aware of which are listed below. In my experience, implementing these would get you on the right track to success.

Note: The list items below are not exhaustive (comments are welcome :-) ) but in my experience having proven this out in the field, its a solid start.

Key Practice #1: Implement a proper Source code management system
Not having the right tool in place spells disaster for the team and for quality which in many organizations isn't such a negotiable quantity. I know this sounds trivial and its assumed most of the time. There are many tools out in the market such as subversion(svn), CVS, Git, clearcase(which has a very high overhead), perforce(uses change sets, an interesting concept). Pick a tool that meets engineering goals of branching so you can have branching capabilities to run current and future releases. If you have a distributed team, pick a tool that supports that easily.

Key Practice #2: Implement Design and Breakdown into manageable parts
Letting the team go off and start coding in parallel is a recipe for potential disaster and will impact your project. Implement the ability to create simple designs so that the teams can understand dependencies up front, be able to have a conceptual, high level approach to coding the feature set. I think there are key areas that should be looked at first like the data model or Domain Object Model( DOM) and together with re-factoring (Key Practice # 4), be able to make changes along the way. The KEY here is to get agreement within the team on the broad concepts. This allows everyone to speak in the same terms.

Key Practice #3: Continuous Integration and Builds
The pre-requisite here is Key Practice #1 since Continuous Integration(CI) assumes a good SCM behind it for check-ins and versioning. I have personally used Cruise Control(with ANT,MAVEN) and Hudson and there are other tools available. I cannot stress how important this is, since the CI system will also run Unit test suites which help with regression testing and help push quality up the chain and early in the cycle.
I have seen in previous projects where the Quality Assurance teams have saved up to 30% time on cycles as a result of implementing this. This also means the engineering team needs to stay committed to building and checking in Unit tests frequently.

Key Practice #4: Re-factoring
One cannot expect to know all the design and coding issues up front since we are not implementing waterfall and even in waterfall, you cannot possibly anticipate all the issues up front. I think that re-factoring must be embraced as standard practice in order to get the quality you need for a winning product. Re-factoring basically means that the functional behavior of the module is the same yet, the code behind it has changed. It has changed because we want to close the technical gap, assumptions,error handling or other aspects , which if not done, create a quality problem. The added benefit is that engineers will get a better sense of how things work as the project moves forward and sometimes assumptions made early are invalid may impact the project. This allows engineers to re-look at code and reduce the impact of change for later modification. Let me clarify what I mean, which is, that if refactoring is ignored, then one can have technical limitations within the code which later on, makes it much more time consuming to change since additions feature/function has been added.

Key Practice #5: Automation/Automated Testing Suite
You have implemented a CI system already, having reached this point, so why not implement the the ability to run Unit tests, BIT (Build Integration) tests, or "Smoke" tests in order to ensure builds get to a stability point early. The point of reaching the best quality, is to push Quality Assurance(QA) up the chain to all levels. I have seen from working on product releases, that one saves a great deal of time within the QA organization from having to run functional tests when there is automation in place. You find that with time, your regression test suites mature, more tests are added, freeing up QA to focus on finding the hard bugs. Ask any QA engineer and they will tell you that functional testing takes a long time so anything that can be done to help your team mates is a good thing.

In addition to the 5 practices I mention above, the final one, which is not to be ignored, but I assume it to hold true is that there needs to be Ownership and Accountability in the team. If there is an problem and we need to resolve it, its important to implement a culture of collective ownership of the code. I was in a situation where there was one key engineer owning key areas and it created a bottleneck for progress, so make sure you build a redundancy model where more that one person knows the particular code line to be able fix issues. This is less of an engineering operation issue, but more a Team issue and a culture within the team which should be instilled early on.

I strongly recommend you implement these techniques and spend time on each of them so that you are able to get the best results in the end. If there are more details you wish to know, please let me know.

Absorption of SCRUM - valuable lessons learnt

2011-03-23T18:58:00.000-07:00

If you are software professional like me and have experience with SCRUM as well as waterfall , yet inherit a team that hasn't had much experience with SCRUM. What should you do as critical success factors to get going and move the team in the right direction. I am reflecting on some recent "real life" experiences that I hope will be useful to you and the community out there. These experiences come from a recent implementation of SCRUM at a fortune 100 software company.

Here is a typical real life situation:

you have a release to push out and you're responsible for it
you have a new team, which has little or no SCRUM experience
you're in charge of figuring out what process is best while not hampering the team and yet instilling a sense of discipline with a new process and there isn't too much time for training

Faced with this challenge I decided that having the right SCRUM Toolkit was an important first step. There are a couple of very good options out there and here are some examples:

SCRUMWORKS by Danube (http://www.danube.com/scrumworks) - there is a free version and a license version. The free version allows you to do basic management like manage sprints, add backlog items to releases, allow a team definition and show basic burndown charts. It works over the internet and its free and a great tool to get started. The paid version has better charting, integration with JIRA and other useful features.
RALLY by Rally Software(www.rallydev.com) - this is a paid for monthly scrum tool, more advanced than SCRUMWORKS, but it depends if you have the budget to cover costs. its free for up to 10 users. For more details on pricing go to the company website. Rally has many good features on tracking and is very comprehensive.

There is always the "excel" version to use which means that an excel sheet is used to look at burndown, tasks left, backlog etc. I am not a big fan of this approach because its hard to coordinate among the team members and if you are managing a larger team, the updates can get time consuming.
Note: There are many other tools in the market, so make sure you use the best tool for your level of maturity in SCRUM and team culture.

Second, I spent time providing my team with basic training on principles of SCRUM, I focused on a couple of key aspects:

1) SCRUM is bottom up - team members should be accountable for hours and there is always a "goto" person for a task to be completed

2) SCRUM collaboration - this really means, you need to ensure your team is open about communications and having the daily scrum. Ensure a team lead, or manager allocates a SCRUM master and ensure daily SCRUMS occur. Ensure this is also monitored and ask only 3 questions:

What did you work on yesterday ?
What are you working on today ?
What is preventing you from reaching your goal ?

The KEY is to not let the SCRUM turn to a long meeting where people try to resolve the impediments facing them. I can't stress this enough, as I find with new team members and people adopting SCRUM, they have a natural tendency to resolve the issues in the SCRUM itself

3) Plan , Plan and Plan - you have to spend time planning and estimating your sprints and how many you will have. Its worth the extra time to do this up front and spend time with the team on this.

4) Burning up, Burndown - the team naturally seems to understand the concept of burndown, but seems to resist the concept of burnup or they don't understand it. It takes time to understand and internalize these concepts. Spend time explaining and guide the team to ensure that its "ok" to burnup if there is legitimate reasons to do so.

5) Backlog management - features go on the backlog, organize them in a release which allows people to frame key features for the release you are currently working on. Explain the difference between tasks and the backlog. Assign level of effort points to items. I typically use Fibonacci series ( 1, 2, 3, 5, 8 ...). For a great article on this read: http://kenschwaber.wordpress.com/2011/03/11/planning-poker/

6) Team updates - its critical that the team update impediments if they arise, close them out and update hours ( burndown ) on tasks. People can think its a "hassle" or unimportant - but its the essence on getting metrics. As a spent a lot of time ensuring updates where timely and accurate.

7) Definition of "Done" - Get proper agreed definitions of what the definition of "Done" means for the team so that when a team member marks a item as Done, there is no ambiguity about it. This is impact to quality further down your cycle if this is not agreed to and followed by everyone.

Thirdly, I was guiding the team and monitoring the process on a frequent basis. What I mean is to facilitate the team in grasping and internalizing SCRUM concepts like daily SCRUM meeting, updates, logging impediments, burning down etc. I think this is key to success when you start, since the team is still absorbing the concepts, working them out between themselves and trying to satisfy software release goals. The amount of monitoring, which is why I like more automated tools, which allow you to manage a "dashboard" is key in real time. This is even more important when you have a distributed team in various time zones ( which was my case).

Finally the last part of the puzzle, is to do a retrospective. I had some spot checks during my release but spent the "downtime" after my release going through completed sprints and what worked and what didn't. We looked at sprints that we planned well and others that we had to invalidate and start over . We also had a frank open discussion about it. As I expected, there were still lingering questions on how to handle certain situations.

SCRUM Maturity

2011-03-23T17:00:00.000-07:00

SCRUM maturity is an interesting concept since its closely related to the complexity of what needs to be done and the number of people involved. When one looks at small teams and new implementation of SCRUM, its important to get to a "beating rhythm" first before attempting more mature concepts like Scrum of Scrums or release road maps.

I like to call the beating rhythm or basic flow, the first step to success and getting it done in a predictable fashion. What I mean is that the team needs to get used to the concepts which are basic and implement these in a predictable fashion and get product out. As teams become better at it, this becomes faster and more reliable.

Basic concepts and tools for starting teams to flow is depicted in the table below:

Documents	Product Backlog (includes features and other log items) Sprint Backlog Burndown Chart
Ceremonies	Sprint Planning Daily meetings Sprint Review
Roles	Product Owner Scrum Master Team

The next level of maturity involves taking the natural flow and doing more complex upfront planning at various levels and moving the concept of Agile up the chain. Moving it up the chain, means making Agile part of the CxO suite and be able to make decisions quickly so that the entire team and company is able to make decisions quickly. This is all about making the product a winner in the market.

There are many companies out there using SCRUM and lots of technology built, but those companies that are able to respond to change quickly and marry the ability to change product to meet market needs are truly the ones creating a winning company with a winning product.

In order to do this the table above changes to add some additional aspects as follows:

Documents	Release Roadmap Product Backlog (includes features and other log items) Sprint Backlog Burndown Chart
Ceremonies	Multi-level planning across departments Daily meetings Sprint Review Scrum of Scrums ( including cross functional disciplines)
Roles	Chief Product Owner/Officer Product Owners Chief Scrum Master(“Uber” Scrum Master) Scrum Masters Team

Agile development at this level involves a high level of interaction between product owners and the chief product officer, as well as scrum masters and the chief scrum master.