First, it is ridiculed,
second, it is violently opposed,
third, it is accepted as being self-evident.

- Arthur Schopenhauer

Our society has passed through thirty years of tearing down established wisdom.  We’ve witnessed a housing bubble where the traditional thinkers were ridiculed.  We’ve witnessed a stock market bubble where the traditional thinkers were ridiculed.  We’ve witnessed many business practices embraced where the traditional thinkers were ridiculed.

We’ve witnessed long established truths replaced with popular theories.  They seemingly worked for a while, which reinforced their support and broadened the appeal, but as they were working, they were sowing the seeds of their own destruction.

Well, here we are with the aftermaths of a stock market bubble and a housing bubble.  Traditional truths that were formerly ridiculed are suddenly accepted as being self-evident.

Oh, the stages truth must pass through to become accepted, and oh, the damage wrought until then.

“Jack Welch, who is regarded as father of the “shareholder value” movement, has said the obsession with short-term profits and share price gains that has dominated the corporate world for over 20 years was “a dumb idea”. ”
– Jack Welch

Jack goes on to say,

“On the face of it, shareholder value is the dumbest idea in the world,” he said. “Shareholder value is a result, not a strategy…your main constituencies are your employees, your customers and your products.”
– Jack Welch

Read the Financial Times article Welch rues short-term profit ‘obsession’

Corporate obsession with short-term profits has been a contributing force behind the adoption and advocacy of the Agile Software practices.  As I’ve argued repeatedly, customer satisfaction and product success isn’t about process or delivering faster, it’s about creating a great user experience. 

Customer value is a result of building a great product; building a great product is the strategy; customer satisfication is the by-product.  In fact, short incremental releases often result in less customer satisfaction, not greater.  

Customers are not the software team’s QA department or product marketing team.  

If you need the customer to tell you exactly what to build, you aren’t the team to deliver on outstanding customer value.  If the customer needs to be a constantly contributing member of your team, you DEFINITELY ARE NOT building the next thing. 

Great people committed to teamwork build great products.  Teamwork is the only time when it’s possible for 1 + 1 to equal 3.  Great teamwork is the only instance when the whole is greater than the sum of its parts.  While individuals are important, the team is most important.  Good process is a manifestation of GOOD TEAMWORK.  Agile’s emphasis on individuals is misplaced. 

Our culture’s obsession with individual’s interest over society’s interest is what brought us to this inauspicious moment in history.  Perhaps we can once again embrace the principles and values that run throughout every modern religion and put the interest of mankind first. 

We do this in business and software development by putting the interests of the company, the shareholders, the customers and the team first.  We lost those priorities along the way with our avarice and obsessive focus on stock options, the stock price, bonuses, and empty careerism.   Where’s the “I” in teamwork? It isn’t.

I’ve been away from blogging on software while I’m busy with another project that unfortunately is consuming all of my spare time.   I couldn’t resist commenting on this article because it rebuked the culture that brought us all here to this moment.  We get angry at Wall Street for taking huge bonuses while their companies are sinking and receiving bailouts from the taxpayer, but we’ve all been part of that culture of excess and greed.  It’s time to begin heeding the words in the Michale Jackson song:

If You Wanna Make The World A Better Place
(If You Wanna Make The World A Better Place)
Take A Look At Yourself, And Then Make A Change
– Michael Jackson, Man In The Mirror

It’s nice to see that “Old School” Values are making a comeback.  Too bad, we have to pay such a painful price to learn what we should have known.   The Agile movement had one thing right: CHANGE IS GOOD. It’s time to CHANGE! This is our wake up call.   WAKE UP!

In the first article we discussed the emotional rollercoaster people typically go through when faced with losing their job.  We learned the importance of acceptance to get us out of the past and start our progress towards looking ahead to tomorrow.  We talked about getting focused and goal oriented as a key to helping us move forward. 

We ended the last article with an easy homework assignment intended to breed some positive feelings and increase our overall focus, which was to start checking off some of the items on our personal “Wish I had time to…” list.  Hopefully you took the advice to heart and started working on your list items and maybe even had the joy of checking one or two of them off. 

In this article we’re going to get you started towards getting that next job by defining your career requirements.  This is a critical step towards your overall plan for getting that next job.   All good plans start with good requirements.  Your career plan is no different.

Getting Started

What are your requirements?  Is it strictly getting a job doing what you were previously doing?  Is it doing the same thing but in a different industry?  Or maybe it is doing something completely different in a different place?  Is it time to pack family and home up to move to your dream location?   Are some of your wants non-negotiable while others are nice to have?  What are you’re willing to trade off and what can’t you go without?  These are the important requirements to drive your career plan.

Until you establish your requirements you really can’t begin planning on how to get that new career.  Well you could but chances are you will end up very unhappy and possibly right back here trying to figure it all out again.  There is no better time than now to define  or readjust your career requirements and priorities. 

Let’s take a closer look at the likely important career requirements that will drive your plan.

Location

If you are considering relocation then this should be the critical path to your plan.  Relocation is the most expensive requirement both financially and emotionally to consider, you need to sort this out before you can build a plan.  The reasons for relocation are many and whatever your reasons you owe it to yourself and your family to do some real good homework before picking up and moving. 

You need to find out as best as possible if the target location (village, city or country) is a place you can happily live.  What is a reasonable salary there?  Do you rent or buy?  What neighborhoods are right for you?  It’s time to start searching the Internet about your target locations.  Check the Chamber of Commerce site, just about every town in the world has one.  Do general searches on the city, read online the local newspapers, read school reports, crime data, etc.  You want to get a feel for the place and to start identifying potential neighborhoods to live, the right schools, entertainment options, etc.  Think of your leisure activities that you enjoy today and make sure the important ones are available in your target relocation. 

Investigate the potential employers in those locations.  How many people are employed in your target field in that area?  How are those companies doing today?  Are they growing, shrinking or on the brink of asking Uncle Sam for some TARP money?

Once you have narrowed your search to a few areas it is time to take a road trip.  Don’t confuse this with a vacation.  You’re not off to see the tourist sites and eat in the best restaurants.  Instead you want to see the areas where you’re considering living.   Find a short-term efficiency apartment to rent where you can cook with the groceries you bought locally.  Do not stay at the 5-star resort next to the National Park!  Check out the leisure activities that you deemed most important to you.  If schools are important, then visit them.  If shopping is important then visit the retail centers.  Whatever you deemed important to your everyday living experience, then make sure you see it in your target neighborhoods.  Bring the whole family.  If you love it, but they hate it, then you’re going to have big problems down the road.

If you’re really serious about picking up family and moving then take your time and do it right.  I have a friend who wanted to move to Florida real bad.  After being let go he decided it was time to make that dream real.  He and his wife did all the homework, visited the area several times, found what they thought was the perfect spot.  He sold his house in NY, bought a new house on a canal in Florida, a boat and was ready for the good life. What he failed to do was include his teenage children in his research.  Well his kids hated the schools, hated the heat, and hated Florida.  In less than 4 months time his wife moved back to NY in an apartment with the kids while he stayed behind trying to sell his Florida dream.  True story.

Target Industries

If one of your requirements is to change industries then your plan needs to include some education on your target industries.  This can range from reading books, industry sites, and company sites within your target industry to possibly attending some formal training.  Basically what you need to know to make sure the industry change is something that you will be happy in.  Getting educated on your target industries serves two purposes - it helps you determine if you really want to work for that toxic chemical processing company and makes you better equipped for the interviews.

Also keep in mind that different industries can pay very differently for the very same skills.  Know what your target industry pays for your skills.  Here are some sites to help you find your value in your chosen industry:

• Salary.com
• Acinet.com
• SalaryExperts.com

Target Companies

Do you have particular work environment requirements?  We all do.  Some examples are a bright sunny building; open floor plans; I must have an office; flat management verses a very hierarchical structured; I need to be part of a team; I prefer to work by myself; flex hours; opportunity to work-from-home; I want an environment where everyone is in the office from 9-5 everyday; lots of travel; I never want to leave my desk; I want to work with internationals, etc.  Whatever your work environment requirements are it is important to make note of these and target companies that can meet your personal needs.  You might not get everything you want, but if you can get the more important ones then you will be happier in the long run.

Target Jobs

If you’re considering any type of career change then you need to investigate the range of roles within that new career.  What is the typical career path?  What is the role you want to eventually achieve?  How might you fit into the career path right now using your strengths and experiences?  Is there training that you require to get you started?  Understanding what you want, what you can do now and how you’re going to get from here to there is essential to your planning. 

If you’re satisfied with your current career path, then take a hard look at how you might improve your skills.  Is there a software language missing from your toolbox?  Is there a skill you’re strong at but never did get that industry-recognized certification?  Is there a skill that you’ve wanted to have but never had the time.  Well you have the time now so start investigating what’s missing and how to add it your personal toolbox.

Contracting

This path appeals to a lot of people who either want scenery changes every few months, or looking to gain lots of diverse experience in a short time. There are different ways of doing this.

You can either hang your own shingle out or work for a contracting firm, where the firm finds you your next job.   Don’t underestimate the time and effort involved in finding your next contract.  People who contract on their own typically estimate that each new contract can take 3 to 6 months minimum to nail down.  Some contracts can take far longer.  That is a lot of lunches, business meetings, etc, without getting paid.  And unless you don’t mind taking long vacations between contracts, then you’ll be doing this while working fulltime on your existing contract.  Some people thrive in this.  Some others fade quickly and return to full time employment.

Talk to people.  We all know contractors of one flavor or another.  Talk to both types.  Ask them how they like it.  How much effort does it take to get the next contract?  How do they find work?  How do they like working for a contracting house?  Talk and ask lots of questions. 

Contracting as an independent is also a nice way to ease you into running your own business and it is a far less risky option, as your only risk is time and not your life savings.  But if running your own business is your dream, then…

Starting a Business

For some people this might be the time to run their own business.  This can be done either by starting up your own business, buying an existing business or buying into a franchise.  If this is something you’ve been thinking about then you have lots of work to do.  You really need to take your time, do lots of research, talk to lots of people.  This is the most risky of all the career options, but also has the most upside of all options. 

Talk to people in the same business but far enough away you would not be considered a competition threat (tougher if you’re considering an internet business).  Ask them how they got started.  What went well?  What were the early pitfalls?  You’d be surprised how if you approach someone right how helpful he/she can be.

If you’re looking to start your own business from scratch then you will need to create a Business Plan.  Over 90% of all new businesses fail within the first two years.  The majority of these failures can be directly attributed to a lack of planning.

Your business plan should cover topics like:

• Define your product
• What is your market and what percentage of that market do you intend on capturing?
• What is your marketing strategy?
• Who is your competition?
• Define the ‘operations’ of your business
• What is the business projected financials (balance sheet, cash flow, break even point, etc)?

Once you have a plan, run it by friends, your accountant, a career counselor and other people who are running their own private businesses even if not in your area.  The feedback is all good and invaluable.  

If you’re considering buying a franchise or an existing business, still consider doing a business plan as it will help you sort out the positives and negatives of the business.  And most important - get a certified accountant to review the business’ books.  Make sure you know what you’re buying into. 

Here are some sites to help you develop your business plan:

• BizPlanIt.com
• BulletProofBizPlans.com
• BPlans.com

Take your time, be thorough in your planning and you will greatly improve your chances!

Summary

Identifying what is important to you, your career and your family are essential to putting your plan together.  Just like any software project, poor requirements only lead to lots of restarts, delays and potentially delivering the wrong product.

If you need more help in identifying your personal career requirements then try Richard Nelson Bolles’ excellent book “What Color is Your Parachute“.  Whether you’re looking for another job in the same career or a whole new career, Bolles covers the bases in a very easy to read book.  If you prefer more direct interaction than reading, I suggest you attend some workshops on career guidance.  There are several available out there, some even for free.  Start Googling and you’ll be surprised what you can find.

So get your career requirements defined.  Like any requirement list, prioritize them and start considering possible alternatives just in case you can’t fulfill some of them.  Your plan will require contingency and will likely go through some adjustments along the way.  If you can identify possible alternatives to some of your requirements then you will be able to change your plan more easily.

While you work on your requirements I’ll be starting the next article on putting it altogether in a plan.  And don’t give up working on your personal “Wish I had time to…” list.  Checking those items off is very rewarding.

In this article I hope to help you get started on the path to your next job by helping you recognize and understand the emotions you are going through.  In future articles I will start developing a personal plan to recovery, examine career alternatives, and explore aspects of finding that next job, but for right now let’s deal with the mental turmoil of what you’re going through.

So you’ve just been let go, now what?

Maybe you had an idea it was coming or maybe it caught you totally off guard (”The guy down the hall for sure, but never me.  They could not afford to let ME go!”).  However it went down, you are likely to go through a rollercoaster of emotions and confusion.

The most important thing to take away here is that there is no shame in receiving that pink slip.  Corporate America has drastically changed over the last decade and a half.  We’re no longer a family with our employer. Instead we’re all just faceless corporate ID numbers today.  When was the last company picnic you attended?  Was your spouse or date allowed to accompany you to last year’s holiday party?  Pink slips are handed out across divisions, groups and by position.  It is very rare that they are personal.  Getting past the perception it was personal goes a long way towards your healing and starts you moving forward.

Emotional Stages

Let’s explore the emotional rollercoaster that you’re going through.  If you can recognize the emotions and the stages they represent, then hopefully you can navigate through it more quickly while minimizing the pain. 

When faced with any life-changing event (e.g. divorce, death of loved one, loss of home, or loss of job) our foundation is rocked sending us on a difficult emotional ride. Although job loss is considered one of the biggies, it isn’t as severe as dealing with the emotions of terminal illness and death as described in the Kubler-Ross Model.  From an analytical point of view the emotional stages of Job Loss extend beyond the Kubler-Ross Model and into positive emotions as we start working on our future. In this article we are only focusing on the emotional strains caused by job loss.

Humans tend to move between three emotional states: reliving the past, confusion in the present and anticipation of tomorrow. 

Living in Yesterday

In the Past state you feel the negative emotions of anger, denial and sadness.  You tend to dwell on what happened, what you might have done differently to prevent it and sometimes even believing that it will all be magically undone and you’ll be called back to your job any day now.   It is hard to leave this stage and start the transition until you accept your situation, leaving the past in the past by putting aside all those negative feelings.

For those who knew the pink slip was on its way, you likely already went through this stage when you first got the hint you were on the short list and are hopefully now well on your way to transitioning to later stages.  But for some it doesn’t sink in until you get the actual paper confirming what you already knew and therefore still have some more time here as you progress towards acceptance.  For those who had no idea the time in this stage is longer as you navigate through all those negative emotions.

The important thing to take from here is Acceptance.  Once you are able to accept your situation you can begin to heal and move forward.

Anticipating Tomorrow

At the other extreme of the past stage is the future stage where you are upbeat, focused, and optimistic. You look forward to what tomorrow will bring.   You might not feel all these emotions initially but the further into this stage you go the more positive you begin to feel.  The entry criteria to this stage are focus and getting goal oriented.

This is not to say that this stage is all joy and bliss.  There will be times of anxiety waiting for the phone to ring for that interview or opening up your email in the morning hoping that job you applied to has finally replied.  And then there are the worries of managing your dwindling finances without knowing how or when the money will start coming in again.  These anxieties will happen, however the difference between now and previous stages is the underlying positive feelings within you. You’re looking forward to tomorrow and the excitement of what awaits you.  There is focus in your life.

Stuck In Between

In between the past and the future stages is generally confusion.  Let’s call this the present state.  You’ve let go of the past but haven’t started really focusing on tomorrow yet.  You’re just kind of stuck in the middle.  For many a lot of time is spent here.  At this point you have learned to accept your loss and have let go of any anger or ill will you might have had.  You know you need to move on but you just can’t seem to get started.  Whether it is doubt, lack of confidence, being out of practice with job search or undecided on what career is next, but whatever reason, you just can’t seem to get going. 

Being unsure what to do next or even how to go about deciding what to do next gets humans very stuck.  We tend to have unclear goals other than “I need a job”.  We feel a bit scattered brain and possibly a bit of anxiety over what lies ahead for us. 

To move on from this stage you need to start gaining some focus.

Transitioning Through the Stages

Most people tend to move back and forth between the 3 stages, especially between the adjacent stages (past & present or present & future).  Be assured that there will be days when you go to sleep excited and ready for the future only to wake up the next morning feeling like you’ve taken two very big steps backward to those very early destructive feelings of the past.  These fallbacks tend to be brief and you will get yourself back on track much faster than your previous stay in the earlier stages.

If you find that you can’t seem to get out of that negative past state of mind or feel endlessly lost in the present state of confusion, or maybe you think you’re ok but your friends and family are hinting to you otherwise, then reach out and seek some help.  There are lots of professional services for career and grief counseling.  Use your favorite search engine, you will find lots of help out there and some of it free through local government, religious organizations and other non-profit groups.   Also try reaching out to friends that have gone or are going through the same experience.  Ask them “How did you get started?  How did you keep yourself busy?  How did you get your Mojo back?”  Talk to people, you’d be surprised how much people want to help you and how much good advice they have to offer.

Homework

To help you get through the emotional stages you have a homework assignment.  I can hear you yelling “Homework?  You’ve got to be kidding, I need a job, not homework!”  We’ll start working on the job in the next article, but for now you need some things to do to help you through the emotional rollercoaster and get you in the right frame of mind to reestablish your career. 

You need some personal non-career goals to get you going.  Even small personal achievements will promote positive thinking, helping get you in the right state of mind to find that next career and job.  Besides, you might never ever have so much free time again.  Take advantage of it. 

It’s time to pick up that musical instrument again, take up that foreign language you always wanted to learn, help coach your child’s soccer team that you just never could find the time to do, DIY around the house. You know what is on that “wish I had time to…” list.  Well it’s time to start checking off some items on that list.  You are going to find checking those items off the list rewarding in more ways than just getting them done.

Also add some regular physical activity to your routine.  If you’re not the athletic type or exercise is a four-letter word to you, well then start with simple walks around the neighborhood or the local shopping mall.  Whatever your current exercise regimen is, it is time to try and up it just a little.  If you’re unsure about your health then see your physician to help build an appropriate exercise plan for you.  Feeling better physically will go a very long way towards a brighter mental outlook.  And who knows, maybe this will be the start of a new buff you.

Another thing to do is de-clutter.  Clutter around you only promotes feelings of disorganization and confusion.  You will be very surprised how cleaning out the closets, straightening out the garage, throwing away the unneeded junk in your basement, reorganize your dresser drawers, all helps you feel more organized, goal driven and thereby giving you some focus and feelings of accomplishment. Try it, it does really help and it is rewarding too. You have the time.

In Summary

To move forward you have to remove the personal feelings around your job loss.  Once you’re over that hump and all the negativity it promotes then you can more easily start moving forward.  To help navigate the emotional stages reach out to friends and family that have gone or are going through similar experiences, and if necessary seek professional help.  Finally in order to help you feel more positive about yourself and to start building some focus in your life you’re going to start checking off those items on your “Wish I had Time to…” list. 

While you’re working on these items I’ll be readying the next article on planning your next career where we’ll talk budget, career options and planning guidelines.  Until then help out at your kid’s soccer team, clean out your closet and take that old guitar out of the closet - dust off those rusty strings one more time and make them shine.

As we saw in the last essay, the 80:20 rule seemingly appears in many circumstances.   When I was attending college and working as a programmer during the 80’s, there were some commonly accepted tenets that guided our software development processes and behaviors. 

• 80% of development effort for a product is spent in maintenance, while only 20% is devoted towards delivering the first release.
• It is many times more costly to remedy a defect in the later phases of the SDLC than in the earlier phases.

We didn’t question these beliefs much because they were commonly accepted as true.  Our own experiences supported these beliefs as well.  I’ve spent 80+% of my career maintaining software while the amount of time devoted to delivering a first release has been small.  I bet most of you reading this can say the same thing.  Entirely new product development, where you start from a blank slate, is rare in this business.

80% Maintenance

Think about it: how often do you purchase an entirely new application?  Most of my software purchases are for the newest release of a product that I already own.  Maybe I’m a Troglodyte, but I haven’t purchased an entirely  new application  in many years. I have, though, upgraded to the latest version of many applications within the last year.

In the enterprise space, many projects are about upgrading to the latest version of the Windows OS, Oracle Database, Apache web server, or ERP application, to name a few.  Even in the software product development space, less effort is devoted to rolling out entirely new tools and libraries than effort devoted to upgrading to the latest version of tools and libraries that you already own and use.

Some may say yes, but being first in a space has many advantages.  Does it?  Who remembers VisiCalc?  Turbo Pascal?  Netware?  Netscape?  CPM? There are litanies of firsts that are distant memories.  There’s an advantage to being second in a space.  Google in Search, Microsoft in browsers and office applications, and Dell in Personal Computers are a few examples where being 2^nd, 3^rd, or later to market was not a disadvantage in their eventual success and market domination.

Realizing that 80+% of development effort is devoted to maintenance, the old school developers organized their processes and priorities accordingly.  Being first is rarely the most important objective.  While being first has it’s virtues, dominating your market is a higher priority, and too often the compromises made to succeed at being  first compromises your ability to dominate the market.

The Old School Manifesto

Here’s what the old school programmers and leaders emphasize in their application development efforts:

• Creating a lasting architecture over quick construction of the first build. When done successfully, this saves time, improves quality, and improves time to market not only for the first release but for each successive release.
• Understanding the customer value over doing what the customer tells you. Very often, real customers don’t know exactly what they want nor do they know what’s possible. Customers are sometimes too busy to tell you what it is that they want. It is the development team’s responsibility to understand the need and to use their knowledge and intelligence to create a superior solution. If you’re expecting your customers to tell you everything, you aren’t the team to be building the next thing.
• Emphasizing the up front work over constructing something quickly. Devote 60% of your effort to the upfront work, requirements and design; because when you do it successfully, the last 40% is a no brainer.
• Thinking before acting over acting before thinking. God gave us intelligence for a purpose. Understanding, prioritizing, strategizing, and planning support success.
• Analysis over just-jump-right-in-and-do-something. It always saves time, money, and heartache.
• Believing that if an error happens in the lab once your customers will see it many times, over rebooting the machine and hoping it never happens again. I know; many defects are difficult to analyze and fix. I’ve worked through many nights addressing difficult defects. Just do it. Eventually difficult defects become easy to analyze and fix, but first you must invest to get there.
• Doing superior work over doing just enough. Superior work inspires good people, and motivates not so good people to perform better. I know just enough is fashionable today, but fashionable is just a passing trend.
• Quality over hitting a date. If quality made Japan the top auto manufactures of the world while being 2^nd to market, it must be telling us something. Listen carefully!
• The future over the present. Developing a product is an investment in the future. Investments are rewarding when good decisions about the future are made in the present.
• Change management over change anytime. Change is a fact of life. We get old; we move; things change. Making good decisions about change is necessary for repeatable success. Most changes are not reactions to changing markets but reactions to the leadership’s change in thinking. This too often reflects inadequate analysis.
• Teamwork over individuals. Individuals come and go in an organization but teams remain. Teamwork is the hallmark of a successful product team. Process is a manifestation of good teamwork. Good teamwork is greater than the sum of its parts. It’s through teamwork that superior products are made and delivered. Value the team and the process that supports teamwork.
• Value over commitments. The ultimate goal of every project is to deliver value. Meet your commitments without delivering value, and all you’ll have is a failed project.

Think About It

It’s become fashionable today to live in the present: just look at the debt our society carries from corporations, to individuals, and to governments.   All are carrying unprecedented amounts of debt.  We act and live today like there will be no tomorrow or that tomorrow doesn’t matter.   Would the worldwide financial crisis have materialized if corporate leaders concerned themselves with tomorrow?  When you concern yourself with tomorrow, the present is normally better as tomorrow unfolds into the present.  At the root of old school values was the idea that we are all working to build a better tomorrow?  It’s time that our culture reaffirms that commitment.  Don’t you think?

What would you add or change in the old school manifesto ?

1. If you are following one of the Agile methodologies, are you satisified with your results?  If not, where is it failing you?
2. What do you find to be the best aspect of the methodology that you follow?
3. If you could change one thing about the way your team practices software development, what would you change?

• 80% of the sales are closed by 20% of the salespeople.
• 80% of the code on a project is written by 20% of the staff.
• 80% of the processor execution time is spent in 20% of the processor’s instructions.
• 80% of your customers use only 20% of your application features.
• 80% of your defects are in 20% of your code.

Erik Peterson presents on the 80:20 rule at a Google Talk, which you can view on YouTube.com.  He explains how to use the 80:20 rule to improve the quality of your testing with the benefit of delivering high quality software products.

History

The 80:20 rule was first demonstrated by Joseph Juran.  The 80:20 rule says that 80% of the effects come from 20% of the causes.  It’s an indicative trend supported by observation and experimentation.

The principal was coined by Joseph Juran, a management consultant notedfor his focus on quality.  He named it after an Italian Economist, Vilfredo Pareto who observed that 80% of the wealth in Italy was owned 20% of the population.

IBM was able to use this insight with great effect when they found that 80% of the execution time was spent in 20% of the instructions.  With this information IBM optimized those 20% instructions giving them a competitive performance edge.  Apparently IBM treated this as a closely guarded secret.

What exactly is it?

Mr. Peterson explains that in Cricket the sweet spot on the bat is where the minimum effort maximizes return.  It’s that small area, the 20% area, on the bat that will catapult the ball the furthest with minimal effort from the batter. It’s what makes professional athletes as they are able to connect the ball with that small area of the bat more often than the amateur.  Identifying the sweet spot, that small area on the bat, is what Pareto Analysis is all about.

Pareto Analysis is the study of the main causes/classifications of something to link them to the properties in the results.  The objective is to establish the relationship between causes and results.

7 step process

A graph is used to perform Pareto Analysis.  There are 7 steps performed in the analysis to identify the most important causes.

1. Build a table listing the causes and their frequency as a percentage.
2. Arrange the rows in the decreasing order of importance of the causes (i.e, the most important cause first).
3. Add a cumulative percentage column to the table
4. Plot with causes on the x-axis and cumulative percentage on y-axis.
5. Join the above points to form a curve.
6. Plot (on the same graph) a bar graph with causes on the x-axis and percent frequency on the y-axis.
7. Draw line at 80% on y-axis parallel to x-axis. Then drop the line at the point of intersection with the curve on x-axis. This point on the x-axis separates the important causes (on the left) and trivial causes (on the right)

Example Behavior

How many Peanut characters did Charles Schulz introduce into his Peanuts cartoons? A survey was conducted where people where asked to name their favorite Peanut character.  The results of the survey unsurprisingly followed the classic 80:20 rule where 80% of the total favorite votes were for 20% of the characters: Snoopy, Charlie Brown, and Linus.  The answer to the beginning question is that there were 54 named characters in the cartoons.

Agile and the 80:20 Rule

Conceptually the Agile paradigm is based on the 80:20 where the objective is to deliver the most important features first. If those features are identified well, it is likely that only a small percentage of the universe of features will satisfy 80% of your customers.

I plan to discuss more about the Agile relationship in a future posting.

Testing and the 80:20 Rule

In 1976 Glenford Myers recommended writing test cases before you execute them, and he observed that defects cluster.   Based on this observation, he recommended pausing what you are doing and writing more tests for where you are finding bug clusters.  In addition he found the following:

• 80% of the defects come from 20% of the modules.
• 50% of the modules are defect free at system test.
• 10% of the modules are responsible for 90% of the system downtime.

Implications

1. Defects are where you are finding them. If you are finding defects in an area, there are more there.
2. Detailed tests are not required for all functionality.
3. Sweet spots are where a majority of the defects are. Find the sweet spots, develop more tests in those areas, and test them more.
4. 10% of the modules are responsible for 90% of system downtime.

As Beck would recommend, look for code smells.  Bugs are where you smell them.

How to Find Code Smells?

Unfortunately, you won’t find the Pareto relationships until you start testing.  It’s essentially a feedback loop, but there are ways to be proactive:

• Identify features/functionality by risk and develop more tests there.
• Concentrate your testing on new features or complicated areas.

As any QA professional knows, often developers aren’t very helpful with identifying these areas.  Erik recommends looking at the checkout records.  Where there are more checkouts, the risk of defects is likely higher.

Alternatively, focus on the areas the customers use most of the time and make them really usable.  Most defects will go unnoticed by customers where the user experience is superior.

Prioritize on Feature Use

Test where the users spend most of their time.  Microsoft apparently uses approach to deliver their products, and in my experience with great effect (see Microsoft’s CEO: 80-20 Rule Applies to Bugs, Not Just Features).   I rarely experience a defect in Microsoft’s desktop applications; the operating system is another story, however.

I would sometimes wonder how Microsoft could deliver such high quality in their office products.   In hindsight it’s quite obvious how they achieve it.  I suspect that I’m like most Office users; I use 5% of the functionality 98% of the time.

If they get that 5% correct, most customers rarely experience a defect.  Interestingly, when I think of the times where I did notice a defect in the Office products, it is when I use a tangential feature like mail merge in Word.

Summary

The 80-20 rule, when applied to testing, can have great effect on your quality and user experience.  In product delivery environments where there is constant pressure to deliver faster, applying the 80-20 rule helps to prioritize your efforts and makes it possible to deliver faster while satisfying a majority of your customers.  While our culture has embraced faster is always better, this is one tool to help identify your priorities with improved product success.

Watch the presentation.  It’s very good.

Similar to hardware fatigue, software exhibits fatigue under continuous usage as well.  For example, a memory leak in an application that runs for a very short time wouldn’t have any consequences.  However, after eight continuous hours of usage, a memory leak could render the application in operable.

Objectives of Soak Testing

Soak testing achieves a number of quality objectives by surfacing errors that would normally go undetected with simple usage.  By achieving these objectives, it is possible to release applications with unmatched quality in your market niche.

Memory

A memory leak of 20 bytes per minute isn’t easily detected after a few minutes.  It’s not easily detected after an hour with only a little over 1k of leakage, but after 24 hours, it’s becoming obvious that memory is allocated without being freed.  

For some applications a 24K memory leak over a 24 hour period may be okay.  However, it’s not just the heap that you may be concerned about.  In early versions of the Windows Operating System, the number of windows, brushes, and fonts that you could create was limited to 64K as the User and GDI data segments were limited to 64K each. 

If your application is leaking window handles and GDI handles, your application could exhaust the available memory to create a new window or font long before the heap is exhausted.  Not only does this adversely impact your application, it also adversely impacts any of the other applications running at the same time. 

As I recollect, exiting the application did not automatically free up these objects, so applications having these errors would eventually degrade performance of the operating system, and a reboot would be required to correct the condition.

While this is no longer a concern for the current Windows operating systems, it may be a problem for other third party libraries that you may be using in your application.

Race Conditions

Soak testing uncovers race conditions.  A race condition is an error in a calculation due to the sequence or timing of execution in multithreaded applications.  Continuous execution of an application raises the probability that the erroneous sequence is repeated many times.  Through repeated occurrences, it becomes possible for the software team to detect the condition.

Deadlocks

Soak testing uncovers deadlocks. In its simplest form, a deadlock is a symptom of a multithreaded application where two threads are waiting on each other to relinquish a resource before either one can continue to execute.  Deadlocks often require the right conditions to exist before they surface.   Through continuous execution over long periods of time, the probability of the deadlock condition being established increases.

Uninitialized Variables

Uninitialized variables are another source of unstable applications.  In particular, an uninitialized pointer can reveal itself in many ways.  If the uninitialized value points within the application data segment, using the pointer will corrupt the values of other global variables resulting in strange application behavior.

Sometimes this may manifest itself as a code exception when an unitialized pointer happens to point to the return address on the stack and the value is changed.  This happens to be a more difficult error to analyze.

Or in other instances, the value points outside the data segment and the application will terminate with a memory exception.  This form of a memory exception is actually the easiest manifestation of this problem to analyze and correct.

Performance Degradation

There are many manifestations of performance degradation.  Continuous execution flushes out many of them.  The most common sources of performance degradation are inefficient memory usage, throughput bottlenecks, and inefficient collection management.

• Inefficient Memory Usage - There are a number of ways this can manifest itself in an application: usually it involves a lot of unnecessary copying of data.  In some applications, the amount of memory copied to perform an operation may grow with repeated execution.
• Throughput bottlenecks - In real-time applications where data is constantly processed, data will back up when data arrives faster than it is processed.  If the application operates for only a short period of time, it can easily go unnoticed.
• Inefficient collection management - There’s nearly always some sort of data management in an application that makes the use of collections, whether it’s an array, a hash table, or linked list.  For some applications, these collections may build to appropriately large numbers of elements.  It’s then that you come to learn that the search algorithms or even the collection is the wrong solution for this problem.

Summary

Soak testing is the only reliable technique that I’m aware of for surfacing the class of application defects formerly described.  The objective of soak testing is not to assess the correctness of the application; it’s not to assess whether feature requirements have been correctly satisfied.  The objective of soak testing is surface defects that will impair application reliability.

Include soak testing as part of your test strategy along with quality release criteria to realize a dramatic improvement in the quality of your product releases.

You cannot control what you cannot measure.
- Tom DeMarco (Software Engineer)

How does the software community improve the control over managing our software development projects?  Ken Schwaber in “Agile Software Development with SCRUM” suggests that the only way to control software development with so much complexity and unpredictability is with an empirical process control model.

Tunde told me the empirical model of process control, on the other hand, expects the unexpected.  It provides and exercises control through frequent inspection and adaptation for processes that are imperfectly defined and generate unpredictable and unrepeatable outputs.  He recommended I study this model and consider its application to the process of building systems.

– Ken Schwaber, Agile Software Development with SCRUM (Series in Agile Software Development)

I happen to agree with Ken’s position on the need for an empirical approach to software process control.  While so much of what is offered by the Agile community is presented as new, the basis of CMM(I) is empirical.  Not only does CMM(I) require empirical control of the software project itself, it requires empirical analysis of the process as well.  Metrics are the basis for all process improvement and project control in the CMM(I) paradigm.

Background

It was during my first experience with CMM(I) process improvement in 1998 when the value of software metrics became apparent to me.  It happened quite accidentally.  I was neutral on the process that we developed based on CMM(I), but I was determined to be a good soldier and give it my support; after all, the company had invested a great deal of money in this initiative.  The least that I could do is to try my best to make it work.  And that’s when the light turned on for me to empirical process control.

CMM(I) identifies two requirements that helped me to arrive at this insight: the basis of estimating is to first estimate the size and then to transform size into duration with a productivity rate; the second requirement is to track completed size as well as completed time. 

Before this, I did not have the tools to accurately and reliably understand how well the project was progressing to schedule.

Why Measure Size?

When you contrast completed size against completed time, problems light up like a Christmas tree.   For example, if completed size on a project is 25% and completed time is 75%, the schedule problem is obvious.  When project management is done well, it’s all about frequent inspection and adaptation, and as Tunde explained to Schwaber, this is the basis for the exercise of control over software projects.  

Tracking with metrics identifies incorrect estimating assumptions early, and consequently, it gives you, the project manager, the control to deliver on the release with no changes in commitments even when the project estimates are incorrect.  That’s the power of an empirical process control model. 

My reservation with Scrum is that the fidelity of the empirical model is low.  You’ve probably experienced it on your software projects. One of the developers on an assignment informs the team that they are done with one of the deliverables.  Two weeks later, the developer continues working on the assignment.  The right metrics will confirm when done is done.

Scrum’s low fidelity is a consequence of only measuring effort remaining.  Measuring both effort and size is analogous to measuring distance to a distant object using the parallax method of measurement.  Measuring the base angles of the triangle is only an interesting observation, but when you add the distance between the base angles, you have the missing information to precisely measure the distance to that distant object.  Similarly, when you contrast completed effort against completed size, you have the necessary information to make accurate inferences about your project’s status with respect to schedule.

In this essay, I’ll explain one aspect of an empirical model that demonstrates the 6-point process described in my previous essay “Software Metrics: An Example Approach.”

Project Conditions 

• You’ve estimated 600 defects to find and fix for a six-month project delivery. 
• The project is 8 weeks away from delivery with 600 defects having been found and with only 200 defects left to fix. 400 defects have already been fixed.
• To reserve time for the unexpected, the manager sets a target to complete fixing the 200 remaining defects in 6 weeks.
• There are 10 developers on the project assigned to defect fixing.
• Project history calibrates average defect fix rate at 1 defect per 1.5 staff days.  Just in case this concerns some readers, this average includes defects that take 10 or more staff days to fix as well as ones that take only hours to fix.  If there are enough defects, we only need to concern ourselves with the mean as the Law of Large numbers tells us that the team will perform at the expected value, which is the mean.
• To keep the example simple for illustrative purposes, the total required defects to fix remains constant during the 6 week period, and developers do not introduce new defects with their fixes, and they fix defects correctly the first time.  While this never happens in practice, introducing these variables into the example unnecessarily complicates the presentation.

Apply the Six Step Process

1. Identify what done looks like with an objective measure.
   1. Inform the team that they have six weeks to fix 200 defects.
2. Break the big goal into smaller goals. 
   1. To achieve the release goal, the team needs to fix 34 defects a week. 
3. Give the team physical productivity goals. 
   1. I like to give stretch goals, so I’ll ask the team to fix 40 defects for the week.
   2. It’s also important to give individual goals because team success is based on individual success.  To fix 40 defects in a week, each developer needs to work towards fixing 4 defects for the week.
   3. Here’s an example of what I’ll say to the team:  “I’d like each developer to target fixing 4 or more defects a week. I realize some of you may have a difficult defect and fix only 1 or even none, and others will have very easy defects and fix 10 or more.  We aren’t being measured on our productivity as individuals, but on our productivity as a team.  If you exceed your individual target but we fail our team target, we failed as a team and individuals.  If we all shoot to exceed our target, the team will easily have achieved the required target of fixing 34 defects a week to deliver on time.”
4. Measure physical progress towards done.
   1. At the end of the week, count how many defects were fixed.  For this example, the team was successful in fixing 38 defects in the past week.
5. Communicate to the team the physical progress towards done and what production is remaining to achieve done.
   1. Last week we were successful in fixing 38 defects.  We are ahead of our goal.  We have 5 weeks left to fix 162 defects. 
6. Repeat starting at step 2 until the overall objective has been achieved.

As is often the case in a real project, additional defects are discovered during this 6 week period, and defects are reopened as they are not fixed correctly.   As a consequence, analysis of the metrics will likely indicate that the original goal cannot be met. In this case at step five, the team needs to make some choices: 

• Defer defects to a future release.
• Add more engineers to the project for defect fixing.
• Extend the release date.

Once the team has decided on a course of action, a new goal is established, and the process repeats beginning at step 1. 

Benefits of the Process

• Everyone on the team knows exactly what is expected of them to achieve success.  Consequently, team members focus more acutely on their deliverables for the week and guard themselves from needless distractions during the day.  People are more productive when they know what’s expected of them.  Having measurable targets enhances communication of expectations.
• Team members are more satisfied in their work.  Having clear and specific goals is motivating to individuals.  Having clear and objective project status is satisfying to the team and to management.
• There’s more collaboration since the only goal that matters is the team goal.  It’s in the interest of team members to help each other to achieve the team goal.
• Status reporting is more accurate, and you can give status with more confidence.  It’s blatantly obvious when productivity is not tracking to schedule.  For example, if only 60 defects are fixed in total after the first two weeks, it’s obvious that team productivity will not complete the work on time unless something changes.
• Once a schedule problem is determined, analysis of the metrics gives you more certainty of what’s required to put the project back on schedule.  Following from the previous bullet, mean productivity for fixing a defect is established at 1.67 staff days.   With that input, the team can accurately decide on extending the target date, deferring defects, adding additional staff, or any combination of the former.

It’s valuable to note when a data trend emerges, it’s rare for it to change - at least not significantly - so don’t expect the mean effort to fix a defect to improve significantly.  Expecting things to change is a sure way to miss the target because when you discover that nothing changed, you will have less time to apply corrective actions.

Nothing Goes to Plan

It is my experience for software projects to always estimate optimistically.  There always seems to be more work than planned, there always seems to be more defects than planned, and there always seems to be more time required to complete assignments than planned.

The strength of an empirical model of process control is when things don’t go as planned.  When more defects are found than planned, analyzing the metrics tells management there is a problem early, the extent of the problem, and what precise options are available to put the project back on schedule.

Summary

To apply the 6-step process successfully, the following data requirements must be met. 

• The size of a deliverable must be estimated and measured.  The more granular the unit of size, the more control you will have over your schedule.  While I have my preference, any unit of size that is measurable works.
• Effort must be calculated from size by dividing it by a productivity rate.

Software metrics are often a controversial subject in the software industry.  I suspect because of the fear metrics will be misused to measure individuals: opposition to software metrics often revolves around the problem of individual productivity.  Software metrics, when used properly, is a tool for measuring the project not individuals; it’s a tool for controlling project deliverables, assessing and communicating status, and making better decisions.