Herding Cats

No Estimates of Costs and Schedule?

2013-05-18T11:44:47-06:00

There is a twitter discussion going on around Neil's post of People Need Estimates. This is a typical agile approach. No real domain or context, just a principle looking for a problem to solve.

First let's establish some ground rules:

I'm a strong advocate of agile development. I work prorgams that use agile. Big programs. Billion dollar programs.
When we are spending other people's money it's not about "us" and our favorite method of spending that money. It's about governance. Don't like governance, then don't work on projects that spend other peoples money.

And the final one ...

Agile - in many cases - is spending people's money to discover the requirements. That's fine. But acknowledge it. Requirements emerge. Requirements change. Requirement are wrong many times. But don't say it'll be OK to use agile and ignore that you're spending - maybe even wasting - other people's money and calling it progress.

Neil says "People need certainty about what they will get and how much they have to spend. " This is not factually true in many domains, not is it actually possible. Those with the money need to know the confidence in the spend plan and the project duration. Certainty is not possible. But don't use that as an excuse for not having a confidence interval estimate:

We will complete this project on or before the 3rd week in November, 2014 with an 80% confidence and a 12% error band on that confidence. This is easily developed and is done every month on the programs we work.
We will complete this project at $1.6M or less with 80% confidence and a 10% error on that confidence.

This is what those spending the money need to know. Otherwise the project is Level of Effort. Work producing good things till the money runs out, get some more money continue doing good things. No problem. That is called maintenance. This is how PayPal uses Scrum. They budget annually for features, develop those features within that budget, repeat forever.

I love the notion "When estimating a date or cost you are creating uncertainty around those things, because you are guessing. "

Well stop guessing. Start being a software professional and start estimating. Have you done this before? How long did it take. Has anyone you know done it before? Go ask them. Guessing is lame. No estimates is even lamer. Worse, it's just being lazy.

One last thing, "People still need 500-page business requirement documents." This is bogus. We work multi billion programs, with 100's of millions of dollars of embedded sofwtare and don't have 500 page requirements documents. Nonsense.

If you don't know where you are going, if you don't have some notion of what done looks like, if you can't tell me approximately how much and how long, please don't start. You can be a binary search process for duration and cost for any tangible piece of software and within five question get without 20%. That's good enough to start. So start and take corrective actions with ne estimate along the way.

Finally "However, I would argue that #NoEstimates gives greater certainty than estimating does." There is absolutely no evidence for this. This is pure speculation and therefore also nonsense. Think about it,

A builder comes to your house for a kitchen remodel.
You tell him roughly what you want done.
You ask "how much will this cost, approximately?"
"Oh I don't know, we're not good at making estimates, let's just get started and see where this takes us."

Are you out of your !@#$'ing mind. It is no different for software. Here's a nice short intro from Target Process, that actually makes sense.

Let's say the CIO comes and says "I need these features", and asks "how much and how long?" You say "Oh we get better outcomes when we don't estimates." Really, and some one believes that nonsense? I guess so.

Quote of the Day - How NOT to be a Risk-Informed Project Manager

2013-05-09T19:49:24-06:00

Yes, this is an actual post on a LinkedIn forum

When the role of the project manager is to push the rock up the hill and then let it roll down again, there is little chance of success, and in the end little need for project management or the project manager. Cancel the project, everyone just go home.

Risk Management is how adults management projects - Tim Lister

Analogies, the Good, the Bad, and the Ugly

2013-05-08T17:59:55-06:00

There have several rounds of how to use analogies and how not to use analogies in the past few years.

These involved notions like agile is an untended garden. Actually and untended garden and called a weed patch.

Or we can't really stop and develop a strategy, because we're always putting out fires. Actually firemen rarely put out fires. They spend the majority of their time preventing fire with fire safety, inspections, fire inspections. Their job is to never have fires start.

And of course for false analogies of the double pendulum as a stand in for chaos and unpredictability. Since the equations of motion are easily defined - an exercise for any upper division physics student - and a MathLab plugin, you can plot the path of the double pendulum.

And my favorite the attractor analogy, in which it is presumed that some how in chaos theory the attractor attracts. Without understanding that those pretty pictures of the attractor are the result of the underlying equations for the system.

So with that in mind, there is a new book from one of my favorite authors, Douglas Hofstadter, Surfaces and Essences. In the book Hofstadter makes the case for analogy as the fuel for creative thinking. Using Robert Oppenheimer's quote...

whether or not we talk about discovery or of invention, analogy is inevitable in human thought, because we come to new things in science with what equipment we have, which is how we have learned to think, and above all how we learned to think about the relatedness of things.

But as always we need to take care to assure that those analogies we use to expand the conversation, don't violate the laws of physics, gardening, or mathematics.

Quote of the Day

2013-05-07T15:35:00-06:00

Everything is simpler than you think and at the same time more complex than you imagine. ‒ Johann Wolfgang Goethe

More Uncertainty and Risk

2013-05-05T14:16:26-06:00

More discussion on LinkedIn about risk, risk management, types of risk, and definitions of risk and opportunity. The "risk management" community in the domains I work is based on Department of Defense, Department of Energy, Department of Homeland Security, NASA, ITIL, AACEI (Association for the Advancement of Cost Engineering International), Nuclear Regulatory Commission, INCOSE (International Council of Systems Engineering), OGC Management of Risk, GAO, and Deloitte's Risk Management Framework.

Several core concepts get confused. So before taking any deep dive into "managing risk," these need to be established.

What is Risk

There are many definitions of risk from a variety of sources. In the end these definitions are all pretty much the same.

Risk involves the probability of something happening in the future.
When that something happens it impacts the project in ways that are not good.
There can be a probability of the effect of the impact as well.
Handling the risk means knowing what kind of risk it is, and what the choices are for handling the risk.

Here's a good definition that can be used in probably any domain.

Risk refers to the uncertainty that surrounds future events and outcomes. It is the expression of the likelihood (probability of occurrence) and (probability of) impact of an event with the potential to influence the achievement of an organization’s objectives. - Managing Risk in Government: An Introduction to Enterprise Risk Management, IBM Center for The Business of Government.

I've edited this a bit to remove the likelihood measure and replace it with the probability just to keep the math straight. You'll see why below when it comes to ordinal values in the risk matrix - a bad idea.

There is a time honored approach to ranking a risk, by calculating some number. This number is meanigless of course, because it has no scale, it is just a ordinal number. We need cardinal numbers for actual risk management.

Risk = Probability of Occurrence x Impact

This is a trick question. The risk matrix approach that results from this calculation treats these numbers as likelihood not probabilities. But in fact they are probabilities. As well they are Ordinal numbers (1, 2, 3, 4, 5 usually) and as such represents the relative ranking of the likelihood of occurrence, not the actual probability of occurrence.

This approach starts with a major problem - a show stopper problem. The two values on either side of the multiplication sign are probability distributions, not real numbers. There is no MULTIPLY operator between two probability distributions. The is the CONVOLUTION operator, but that is not likely to be of much value to ordinary project management staff. One starting point for a better understanding of the problems with the risk multiplier and risk matrix approach is the work of Tony Cox. Optimal Design of Qualitative Risk Matrices to Classify Quantitative Risks. Tony's source paper provides the basis for this discussion, "What's Wrong with Risk Matricies."

To put it in blunt terms.

The combination of Ordinal Scales and Risk Matricies are Fatally Flawed

In this link, it is stated again, so I'll restate it here

It is a category mistake to multipy ordinal numbers

By category error it means the objects, the categories, do not have the proper properties to work with the operators. For some more discussion of this topic and a bit of counter discussion, see What's Right with Risk Matricies.

Where does risk come from?

Risk comes from uncertainty. There are two kinds of uncertainty.

Uncertainty that can be reduced - reducible uncertainty. This is called epistemic uncertainty. This uncertainty is characterized by the lack of knowledge. We can buy this knowledge.
Uncertainty that cannot be reduced - irreducaable uncertainty. This is called aleatory uncertainty. This uncertainty is an inherent variability in the project processes and characterized by a probability distribution. We can not buy more information about this uncertainty.

The critical point here is the reducibility and the irreducibility of these uncertainties.

If the uncertainty is irreducible, then only margin can be used to protect the project from the risk. This is scehdule margin, cost margin, techncial performance margin.
If the uncertainty is reducible, then we can buy down the uncertainty and therefore buy down the risk. That is we can spend more to find out more information - increase our knowledge. This is done in one of two ways:
- Spend money in the project to increase our knowledge.
- Provide a Management Reserve or Contingency to handle the risk when it comes true.

How is Risk Maanged?

We must start with a risk management process. There are several, but they are essentially the same. Here's my favorite, there are others of course, but this one covers all the steps.

So What's the Point Here?

There is no need to make up definitions for risk and risk management. There are plenty around and all of them are in use on actual projects by actual project and program managers, managing risks every day. Providing your own definitions, is sporty at best. At worst, it shows you haven't done your homework.
Uncertainty is the source of risk. There are two types of uncertainty - reducible, epistemic and irreducible, aleatory. If you do not separate these two, you cannot credibly have a risk handling plan. If you do not have a credible handling plan, then you're not actually managing risk.
Ordinal numbers cannot be used to make risk decisions. Cardinal numbers are needed. No ranking likelihood of occurrence as 1,2,3,4,5. Not allowed, it's bogus. Same for impact. Bogus. And certainly no multiplying those two, even more bogus.
Risk Management is How Adults Manage Projects - Tim Lister

Maximizing Project Value - a Review - Chapter 2

2013-05-03T11:00:00-06:00

Chapter 1 of Maximizing Project Value introduced the idea of value. Chapter 2 speaks to where and how that value flows. A couple years ago there was confusion about the term value. Let's restate it here again. The value John speaks to is the business value. He speaks to Earned Value later in this book, but even more so in Project Management the Agile Way.

Chapter 2 Highlights

There is a distinction between project value and business value - project value is measured by Earned Value - how much of your budget did you earn? Business value must be derived from the business strategy. A business case if fine but a measurable strategy is better. Balanced Scorecard is the best way to connect business value with Earned Value. Here's how to do that Notes on Balanced Scorecard.
Interests of the customer must be balanced as the value is developed - one way to do that is with a Scorecard that connects project performance with strategy. Then the customer can see the Measures of Effectiveness (MoE) of the project's outcomes against the fulfillment of the strategy.
Projects are the instrument of strategy - yep this is how strategy gets implemented. The literature shows strategies fail most often during execution.
There is a six steps process from opportunity to projects. Projects drive value - I prefer the Goal Question Measurement approach to making these connections. John's approach is straight forward and simple.

I've found what I was looking for in Chapter 7 that makes the book critical to our success - connecting Business Value with Capabilities Based Planning. I won't skip ahead. For now Chapter 2, is the starting point for putting these ideas to work.

Cost (and Schedule) Estimating Foundations

2013-05-02T09:02:00-06:00

Cost estimating methods have been around for a long time. The current processes found in agile use a points system, sometimes a Fibonacci series to bin the values of the points.

The challenge with this approach is the estimate in agile is not monetized so we can't really tell if the Total Allocated Budget (TAB) is sufficient for the project at any point in time, unless the capacity for and the quality of the ourcomes is steady - that is Level of Effort.

With the LOE approach, the capacity for work is the critical measurement needed for estimating the cost at completion. As well continuous updating of this capacity for work is needed and correctly done on good agile projects.

But there are other issues with this LOE approach on larger projects:

Do we know the variances in the capacity for work and how those variances will impact the final Cost at Completion?
Do we know the interdependencies between the various work products and how they impact the final cost?
Do we know the Aleatory uncertainty - the natural occurrence that can't be fixed and has to have margin.
Do we know the Epistemic uncertainty - the event based risks that need a risk handling plan?

So the examples like that found at Projects @ Work, don't really consider any of the underlying uncertainties in estimating. Without the next level down - statistically adjusted estimates of the work effort, the capacity for work, the quality of that work, and the interdependencies between those work activities and their products, the simple and maybe even simple minded approaches to estimating have limits to scaling.

This is one of those topics where everyone is right in some way, depending on the domain, context in the domain, and scale of that domain. As agile enters the larger acquisition community, where we're spending other peoples money - maybe 100's of millions of dollars, care needs to be taken when applying un-monetized, non-probabilistic, non-joint probability (cross correlations between work element) and non-stochastic forecasting models. The real world is not that simple.

Both Aleatory and Epistemic Uncertainty Create Risk

2013-05-01T08:55:00-06:00

Over the past weeks there have been several discussions on the forums and Blogs amount risk and risk management. Here's a short post on those topics and their impact on project performance.

Risk Comes from Uncertainty

Risk does not exist by itself. Risk is created when there is uncertainty. If I am certain that it is going to rain this afternoon, then there is no risk of rain. It's going to rain with 100% probability. There is no uncertainty about the forecast of rain.

So first we need uncertainty to have a risk. But there are two classes of uncertainty:

Aleatory uncertainty - is uncertainty that comes from a random process. Flipping a coin and predicting either HEADS or TAILS is aleatory uncertainty. In other words, the uncertainty we are observing is random, it is part of the natural processes of what we are observing.

Aleatory uncertainty refers to the inherent uncertainty due to the probabilistic variability.

This type of uncertainty is Irreducible, in that there will always be variability in the underlying variables.

These uncertainties are characterized by a probability distribution.

The parameter that is being measured - duration, RPM, discharge from a river flow, is stochastic and cannot be reduced.

Epistemic uncertainty - is uncertainty that comes from the lack of knowledge. This lack of knowledge comes from many sources. Inadequate understanding of the underlying processes, incomplete knowledge of the phenomena, or imprecise evaluation of the related characteristics are common sources of epistemic uncertainty. In other words we don't know how this thing works so there is uncertainty about its operation.

Epistemic uncertainty refers to limited knowledge we may have about the system (modeled or real). This type of uncertainty is reducible.

If we have more information, we can take more measurements, conduct more tests, "buy" more information. This type of uncertainty can be reduced.

The parameter being measures is usually a characteristic of the material or the physical process. The uncertainty is related to the "lack of knowledge," about this parameter.

Handling Risks Created from Uncertainty

Aleatory risk is not a lack of information. It is a naturally occurring process. We cannot buy more information, so we have to provide margin for this type of risk. Schedule Margin to cover the naturally occurring variances in how long it takes to do the work. Cost Margin to cover the naturally occurring variances in the price of something we are consuming in our project.

Aleatory uncertainty and the resulting risk is modeled with a Probability Distribution Function. This PDF describes all the possible values the process can take and the probability of each value. For a single toss of a coin, there is a 50% probability it will be either heads or tails. For multiple tosses of a fair coin the probability distribution of the total number of heads or the total number of tails is a binomial distribution that looks like this for the numbers of HEADs from fair coin being tossed 20 times.

The PDF for the possible durations for the work in the project can be determined in several ways. It turns out we can buy knowledge about aleatory uncertainty through Reference Class Forecasting and past performance modeling. This new information then allows us to update - adjust - our past performance on similar work will provide information about our future performance. But the underlying processes is still random, and our new information simply created a new aleatory uncertainty PDF.

Epistemic risk comes from the lack of knowledge. Epistemology is the branch of philosophy concerned with the nature and scope of knowledge. Lack of knowledge is epistemic uncertainty.

Epistemic risk is modeled by defining the probability that the risk will occur, the time frame in which that probability is active, and the probability of an impact or consequence from the risk when it does occur.

Risk statements are used to define and model these event based risk:

IF-THEN - says if we miss our next milestone then project will fail to achieve its business value during the next quarter.

CONDITION-CONCERN - our subcontractor has not provided enough information for us to status the schedule, and our concern is the schedule is slipping and we don't know it.

CONDITION-EVENT-CONSEQUENCE - our status shows there are some tasks behind schedule, so we could miss our milestone, and the project will fail to achieve its business value in the next quarter.

For these types of risks we can have an explicit or an implicit risk handling plan. I use the work handling with special purpose. We handle risks in a variety of ways. Mitigation is one of those ways. But the risk handling work is actual work. It is in the schedule. We are doing work to mitigate the risk. We are buying down the risk, or we are retiring the risk. In all cases, we are spending money, and consuming time to reduce the probability that the risk will occur. Or we could be spending money and consuming time to reduce the impact of the risk when it does occur. In both cases we are taking action to address the risk.

The second approach to handling an epistemic risk is the have Management Reserve to cover the cost of the consequences when the risk occurs. Sometimes the term contingency is used. Both Management Reserve and Contingency may be used together. In both cases, money is set aside to handle the risk. We also need time as well, so we may have schedule reserve. But this gets confused many times with schedule margin, but it is still needed.

Risk Management is how Adults Manage Projects

One of the posters stated what would be considered a Lame response to the processes and seeming conplexity of managing risks on non-trivial projects, by stating you're making this to complex - Just Do It. It was lame. Here's the response to those who objective in what ever way to doing risk management.

First answer the question what is the value at risk for your project? Don't know? Go find out. Then ask the project sponsor or the person giving you money to manage the project, if they would be willing to lose that money outright. Just write it off when the risk comes true. Probably not would be the answer. So go do the risk management process.

Here's Tim Lister's advice. The section title is Lister's quote and should be used every time some lame response comes back about risk management.

Maximizing Project Value - a Review

2013-04-30T08:09:00-06:00

John Goodpasture sent me his new book Maximizing Project Value, Management Concepts, 2013. I promised John I'd review the book. I started to write a simple review, but then I started marking up my copy and learning and use new ideas. So here's my new approach. I'll post a talking points review of each Chapter across the next few weeks. This is more than a book review. It is a cliff notes for the book.

Chapter 1 - Understanding the Value of Projects

The project's value is the worth of a project's throughput, or the difference in business value measured before and after a project, as earned during the course of the project schedule.
Managing the tension between business and the project is optimizing project success in the context of business success.
The object of risk management and optimization is to effect the best overall outcome as valued by the business. This is an important concept.
Project managers are leaders as well as managers - this should end the discussion about if PM is management or leadership. Both, everyone back to work.
The value earned is planned as the project's budget. This is the simplest definition of Earned Value.
Eberhardt Rechtin defines emergent value as value created solely by the relationship among the elements of the system. Rechtin's book must be on the shelf of any credible project manager.Systems Architecting of Organizations: Why Eagles Can't Swim and The Art of Systems Architecting, Second Edition. Rechtin is the orginal systems thinker.

Chapter 1 is a great opening, defining how project value is measured.

Have Project "Bodies of Knowledge" Gone off Track?

2013-04-29T08:35:00-06:00

Shim Maron provided a link to a post from APM titled We Need to Talk About Bodies of Knowledge.

Here's my counter. There are Principles, Practices, and Processes that are immutable for all projects in all domains for all "methods" of managing the project. That's why they are called Immutable.

5 immutable principles and 5 processes in 60 seconds from Glen Alleman

No doubt the BoK's have gone off track when seen as "prescriptive" for working projects. But the basis of knowledge needed for success is still there in the Principle, Practices, and Process. A test of this approach is similar to the 12 principles of the Agile Manifesto.

The agile manifesto created a new movement in software development, but lost its way when there were no underlying principles. PM BoK's have lost their way because they don't start with Immutable principles, practices, and processes. They start - both APM and PMI - with prescriptive knowledge areas and process groups which get confused with methods.

I would conjecture that the 5 PPP's have a matching document in the OGC P3M3 which was used in the UK for some time. It is a maturity assessment vehicle, but starts with core principles. None of the BoK's to date start with "core principles," in the way Plato did with "what is a chair."

Learning Is Now More Important Than Knowledge

2013-04-28T09:37:00-06:00

When I encounter a "subject matter expert," I've now learned to assess if this person is a "knowledge holder," or a "learning individual." This is what informed that "learning." This came to my from a friend and neighbor who is a graduate of the University of Chicago in economics and spoke at this conference.

In our business - project and program management - there are lots of "knowledge" providers, assessors of knowledge, frameworks for measuring our knowledge. What is needed now is how to learn to make better decisions about our projects. This is not going to happen without a decision making framework for the core principles, practices, and processes for project management. The current Bodies of Knowledge, self-proclaimed providers of better BOKs and assessment tools, authors (I'm one of them), and experts in the field, all need to reassess what we are doing to increase the Learning abilities, beyond the just the possesion of knowledge and the dispensing of this knowledge to others.

How can we actually learn to improve our probability of succes. The Decision Scientist for project success - as suggested here - needs to have art, science, and the scale to provide actionable information for the decision makers for the project.

We have most of the data we need, what we don't do is make decisions on this data, or even know how to make decisions with this data. We report it, but don't use it to make decisions. Numbers matter, but leadership needs to use the numbers.

Forecasting Future Performance

2013-04-27T05:22:00-06:00

Forecasting involves making projections about future performance on the basis of historical and current data.

The chart below is a synthetic (same as fake) Cost Performance Index at the program level 2½ years. The area on the right is the forecast of possible values of the CPI given the historical behaviour, with tuning parameters and smoothing functions.

The next step is to use real program performance data over some realistic period of time - say 4 year - and use H-W (or a similar forecasting tool), to confirm that a less that complete assessment, say take 2 years of data, forecast a 6 month value, and see of the actual 2 year, 6 month value come out close the to the forecast.

This is all done in R, with very simple commands, once the time series for the CPI is formatted and partitioned properly.

The Myth of "Trying Things Out"

2013-04-26T21:01:00-06:00

There's a tremendous popular fallacy which holds that significant research can be carried out by trying things. Actually it is easy to show that in general no significant problem can be solved empirically, except for accidents so rare as to be statistically unimportant. One of my jests is to say that we work empirically — we use bull's eye empiricism. We try everything, but we try the right thing first!

Edwin Land - The Journal of Imaging Science and Technology, Vol. 37, No. 3 (1992), p. 537

The notion that the architecture will emerge, that requirements appear as we go along spending other peoples money, and the customer knows the right solution when they see it, is just that "notional."

Without some agreed upon description of what Capabilities are to be provided by the system, you'll just be spending time and money to find out you don't know what DONE looks like.

A Breathtaking Paper

2013-04-25T08:39:00-06:00

Came across "Lost Roots : How Project Management Settled on the Phased Approach (and compromised its ability to lead change in modern enterprises)," Sylvain Lenfle and Christoph Loch.

From the abstract:

The discipline of project management adheres to the dominant model of  the project life cycle, or the phased stage-gate approach, of executing  projects. This implies a clear definition of mission and system at the  outset (to reduce uncertainty), and subsequent execution in phases with  decision gates.   This approach contrasts with the way the seminal  projects were conducted that are credited with establishing the foundation of the  discipline in the 1950s. These  projects started with missions that were beyond the currently possible,  thus any solutions had to emerge over time. They succeeded by a  combination of parallel trials (from which the best would then be  selected) and trial-and-error iteration (allowing for the modification  of solutions pursued over a period of time).   Although the success of these approaches was documented and explained by scientific  work in the 1950s, today they seem to fly in the face of accepted  professional standards, making managers uncomfortable when they  encounter them.   The explanation for this contradiction has its roots in the 1960s, when  the so-called McNamara revolution at the Department of Defense gave a  control orientation to the PM discipline. This shift was cemented by the  encoded practices of the DoD and NASA, contemporary scientific writing,  and the foundation of the Project Management Institute as a professional  organization that translated the standard into the educational norm for  a generation of project managers. The project management discipline was  thus relegated to a "grunt work niche" - the engineering execution of  moderately novel projects with a clear mission. As a result, it has been  prevented from taking center stage in the crucial strategic change  initiatives facing many organizations today.   This article describes the historical events at the origin of PM's  reorientation, arguing that the discipline should be broadened in order  to create greater value for organizations whose portfolios include push-the-envelope projects.

Agile may ne making a come back in project management, but the contracting aspects of large programs needs to deal with the uncertainties as well as the developers.

Quote of the Day

2013-04-24T07:59:00-06:00

Terminology that belongs to one knowledge system is often misunderstood by people who work with other knowledge systems. Overcoming linguistic barriers between systems is essential.

"Reducing Natural-Language Ambiguities in Requirements Engineering," Lars Schnieder and Susanne Arndt, Ask Magazine, NASA.GOV, P.38

Averages Without Variances are Meaningless - Or Worse Misleading

2013-04-23T08:10:00-06:00

The current Flap over the Public Debt and Growth paper originally produced by Reinhart, Carmen M, Vincent R Reinhart, and Kenneth S Rogoff. 2012. Public Debt Overhangs: Advanced-Economy Episodes since 1800. Journal of Economic Perspectives 26, no. 3: 69-86, brings out some core issues we face in the management and assessment of complex projects using performance data.

As well, there is a flap over the use of Excel for modeling econometric processes. Which was addressed more in the NPR segment. More on Excel and modeling later.

In the original paper - which has now been updated, there are several key words that trigger red flags in any credible statistical analysis of dynamic systems like the economy or complex programs. There is an assessment of the original paper. Let's start with the response to the original paper from the original authors addressed to the author of the critique.

This brings us to the core conceptual issue, which Herndon, Ash and Pollin argue greatly biases our results. They argue that we (they) use an “unconventional weighting of summary statistics.” In particular, for each bucket, we (they) take average growth rates for each country and then take an average of the result.

Averages without variances are not very useful. We all know the story of measuring the most likely temperature in two locations Cody Wyoming and Trinidad-Tobago and the misrepresentation of Averages.
Since economies and projects are driven by stochastic processes, we must model their data as random.
Everyone claiming to forecast the future or make correlations between random processes must have on their shelf and have demonstrated to have read How to Lie with Statistics, Darrell Huff.

So ignoring for the moment the very naive statement by two Harvard professors of averaging the averages in the absence of the variances, and then making projections of the correlations, there are other problems.

Using Excel for modeling can certainly be done, but there are much better ways to model statistical and probabilistic processes - R for one. MathLab is another.
Excel also suffers from the in auditability problem. If you build a spreadsheet and send it to me to make management decisions, I have to look at every cell to determine the right formula, right range for that formula, and right data is in the right rows and columns for that formula to work. A pain in the ass for simple spreadsheets. A nightmare for larger ones. This is simply bad modeling. Especially since R is free and an academic version of MathLab or Mathematica is under $300. Come on Harvard professors, earn your reputation.

So What Does This Mean for Project Performance Analysis?

There are pictures in the links above to show that projects are statistical processes. So here's the bottom line:

All numbers in projects and econometric models are random numbers.
Correlations between the source of these numbers may also be random or at least changing over time - nonstationary stochastic processes.
Models that don't consider this information are probably not that useful and may actually - and I'll be crass here - BOGUS.
If we are ever to produce credible models of how projects work - cost, schedule, risk, and technical performance models - they must be credible statistical models. That means averages and averages of averages are simply not allowed without the variances and even higher order moments and the correlations between the generating functions of these random variables.

So however, this turns out with the Bad Math people at Harvard, we've got to do better.

The current Earned Value Management processes don't consider the statistical nature of the performance indices. This is bad. Here's a simple and very understandable paper "Performing Statistical Analysis on Earned Value Data."
As well, the calculations of future performance use cumulative numbers which wipe out the varainces, the current period which is a point sample to compute the Estimate At Completion. This is not only bad it is naive math.
And then we are surprised when things don't turn out as expected.

Some final thoughts

Correlation is not causation.
All numbers are random numbers drawn from a known or possibly unknown population. If you don't know the population statistics your assessment of the numbers is different than if you do.
All random numbers need variance, standard deviation, and higher order moments to be considered credible as sources of any analysis.
Drawing a picture of the average of anything is very sporty without the proper statistical analysis.
Read Huff, reread it, read it all the time. Also go buy a good statistics and analysis book. Start with Advanced Statistics Demystified.
If there is an academic paper that is being used for public policy has not been peer reviewed, and then throw it away. The Reinhard and Rogoff paper is the basis of Paul Ryan's economic plan. He may have really good points, but it built on sand.
Same advice from self-proclaimed experts in anything, especially project management. If there is a PhD thesis that has ZERO references in CITESEER, ignore it.
Read more in the linked references below to see how bad statistics can go even more bad.
Download R, download the free books, learn how to think and act with credible statistics. Learn how many sample you need, learn how to assess the needed population statistics confidence levels, learn how to make decisions based on confidence levels not absolute numbers. We have a 70% confidence of completing on or before a date, or we have a 70% confidence or completing at or below of specific cost must be the answers when management asks about cost and schedule performance.

Quote of the Day

2013-04-22T08:07:00-06:00

Every man, however wise, who begins by worshipping success, must end in mere mediocrity - G. K. Chesterson

If you're looking for the sure fire formula for success, be it business success or project success, you're going to be disappointed. There are ways to increase the probability of success. But only increase not assure.

Dan Ward has six suggested ways to success

Naivete - believing something can be done, even if it has never been done before
Recklessness - taking action even when a positive outcome is not guaranteed
Monomania - single-minded dedication to a particular goal
Cheating - doing the right thing even when no one else does
Spying - collecting, sorting, storing, accessing, using, and sharing information

Lt Col Dan Ward is the Test & Evaluation lead for the Theater Battle Control division of the USAF's new Life Cycle Management Center at Hanscom AFB.

Agile and the Federal Government

2013-04-21T09:37:24-06:00

TechWell is an agile site that has been around for awhile. There is recent post on Agile and Federal Government. While the concepts of the post are useful, there are some issue with the underlying processes that need clarification.

Earned Value Does NOT Define the Development Process

The post suggests EVM requires Waterfall. First the development process defined in the picture on Page 2, is not allowed. Not to say there aren't people out there using it. While the Integrated Defense Acquisition, Technology, and Logistics Life Cycle Management System may appear overwhelmingly complex, it is the basis for all complex acquisition processes. It defines an itertaive and incremental process for buying weapons systems. Increasing maturity of the deliverables is defined at each Program Event.

But in an EVM program which is defined by several Federal Acquisition Regulations and the Defense Supplement, DI-MGMT-81866 calls out an Integrated Master Schedule. Along with is the DOD acquisition Life Cycle process where incremental maturity of the delivered products are defined in the Integrated Master Plan. The Guide of the IMP/IMS shows how to build a credible Plan and Schedule for a program.

We have to remember EV is only required on programs greater than $20M and a validated system on programs greater than $50M. DOE requires EV on programs greater than $5M and HHS has similar requirements. But $20M of software development is a BIG program. And you'd better have something better than Scrum looking after the expenditure of $20,000,000 of the government's money.

Waterfall Replaced By Rolling Waves

The second picture suggests that Agile provides an iterative solution. But in fact Rolling Wave development, defined in the acquisition guides does the same. RW's can certainly be a substitute for iterations.

But a better solution is to move the Agile Iteration down to the Work Package level. A Work Package in the Earned Value Management paradigm, is a self contained "package of work" that produces a tangible outcome that fulfills the Criteria needed to complete the Accomplishments that define the increasing maturity of the deliverables for a specific Program Event. Look at the IMP/IMS Preparation and Use Guide for how to do this.

So Here We are Again

I'm speaking again at EVM World in Naples FL, on integrating Agile with EV. Notice a critical phrase, Agile integrated with EV. Not the other way around. If you have a project less than $20M and are currently using Agile, there is no need for EVM. Don't do it, assuming you've actually got control the program. EV can add value for sure, but it's not mandated.

Here's the previous briefing on integrating agile into EV - EV+Agile=Success. The current one is slightly better and will be available May 26th, 2013.

So What's the Point

While there is growing enthusiasm for integrating agile into government programs, the acquisition community has not caught up. The agileist suggesting how to do things need to understand more of how the acquisition community thinks about managing the government's money. Open ended development may be the rage in commercial business (probably not), not if we are be to good stewards of the public's money more than just agile needs to be in place.

This notion of agile on governemtn prorgams is itself an emerging topic, that needs much more discussion before true value can be created. There are examples of success in the AirForce, DOE, and NASA. I've worked some of those programs and written about them in conference proceedings. But to simply apply agile in the absence of the acquisiton communities participation is going to lead to disappointment. Agile is needed, but so are the controls in place in the FAR/DFARS processes.

An Interlude

2013-04-18T08:24:00-06:00

A new release of Sakharov's book that was banded in the Soviet Union, with the content smuggled out on postcards. This book details the search for the structure of matter and the forces that hold this matter together.

For project managers, the process of discovery and theory are tightly intertwined in similar ways to physics. Our theories are relatively simple - some fundamental principles and some fundamental practices and the some several types of processes around those principles and practice.

The theory of project management hasn't really changed that much. What Sakharov speaks about is the abundance of particles and the failure of the theory to explain why. Turns out of course the explanation is the lack of understanding of the underlying processes that produce the particles.

Our notion of the multitude of ways to manage a project have lost the connection with the fundamental principles, practices, and processes. So no matter the details of the implementation of a method, a tool, a cleaver approach, the Principles, Practices, and Processes need to be in place of any hope of project success

Quote of the Day

2013-04-17T08:50:00-06:00

All the world's a System

All the parts are Separate

All the parts are Connected

Successfully managing a project is a systems management process. Project's have parts that are separate - look at the Work Breakdown Structure.

Project's have parts that are connected - look at the Integrated Master Plan and Integrated Master Schedule.

Project's have combinations of separate and connected - look at the Integrated Risk Management processes.

Project Management has Principles, Practices, and Processes. If we don't see the whole and the parts at the same time, our chances of success are reduced.