Chief Financial Officers, Do You Know What is Lurking Inside Your Forecasts?

The well-known Mazlow hierarchy prioritizes human behavior starting with basic survival and ending with self-actualization.  A subset of the human species, the Chief Financial Officer has its own hierarchy of motivations, as visualized below:

First on any CFO’s mind, the organization must have enough cash to meet its obligations to employees, vendors, creditors, and the government.  Many CFOs will conduct internal earnings forecasts to provide an expected cash position. There is a serious problem with this approach. Just because the expected cash position is positive doesn’t mean that there is no chance of running out of cash. This is an example of the Flaw of Averages.

What is Lurking Inside Your Cash Flow Forecast?

Let us illustrate this problem with a scenario. The CFO recently completed an earnings forecast that projected ample cash to meet all obligations including a $50 million debt payment due at quarter end.  The CFO was not informed that lurking inside the forecast was a 9% chance that a large sale would fall through and a line of business would experience severe cost overruns.  While the CFO would rather work on optimizing capital allocations to maximize long-term value, suddenly, the CFO is faced with defaulting on the loan or missing payroll.  Either way, the CFO is out of a job.

Had the CFO been aware of the chances, the CFO could have taken actions ahead of time to strengthen the cash position.  We call these actions the “CFO Levers.”

1. Cut expenses (operating expenses and capital investments)

2. Call in accounts receivables early (at a discount of course)

3. Ask the bank for an increase in the line of credit (LOC)

Suppose the forecast has been completed with the results shown in Figure 1 below:

By quarter end, the forecast projects $60 million in cash and the CFO is assured of a positive cash position even after the debt payment.

CFO, are you prepared for your career ending soon?

Feeling lucky? Don’t put away your CFO levers just yet.

Everyone knows that forecasts are just predictions, and predicting is hard, especially the future[1]. Most forecasts produce a single-number: “We’ll have $60 million cash at quarter end”. That number is a comfortable lie. 

Behind every single-number forecast hides a range of outcomes – some better, some catastrophic.  How does the CFO gain access to this range of outcomes?

There is a solution. The CFO knows from prior experience how badly forecasts have missed once the actual revenues and expenses are known.  AI and other recently developed tools and standards[2] enable today’s CFO to transform static forecasts into stochastic forecasts which, you guessed it, provide the chance of running out of cash by month, as presented in Figure 2 below.

The CFO learns that cash adequacy is not assured and there is an 8.9% chance of running out of cash by the end of the quarter.  Few CFOs will accept an almost 10% chance of a career-ending event.  Time to start working the CFO Levers to avoid a forced early retirement!

Introducing the Cash ChanceOmeter

The CFO’s decision process can be demonstrated through an interactive ChanceOmeter that provides instant feedback on the impact of moving the CFO Levers. This is based the Open SIPmath™ Standard for stochastic data, which allows the results of Monte Carlo simulations to be combined and distributed across the enterprise for use in Excel, Web apps or even on a phone.

For the submitted forecast, based on single-number estimates, the static ChanceOmeter in Figure 3 below displays a misleading 0% chance of running out of cash and no need to make use of the CFO levers. The stochastic results, which quantify uncertainty, tell a very different story.  As we saw earlier, by month 3 there is an unacceptably high 8.9% chance of running out of cash as shown by the stochastic ChanceOmeter in Figure 4 below.

Figure 3. Based on static forecast Figure 4. Based on stochastic data calculations

The ChanceOmeter transforms the CFOs static forecast into a stochastic dashboard.  It shows:

• The actual odds of running short, by month and for the quarter

• What each CFO lever buys in reduced chances of running out of cash

The CFO activates the levers and instantly learns from the ChanceOmeters in Figure 5 below. The CFO understands that there are long-term costs to these levers – foregone future revenues, higher debt service costs, and impacts to employee morale and productivity – and must assess the trade-offs carefully.

• Reducing or delaying expenses by $10 million is a powerful lever, reducing the chances of running out of cash by more than half, to 4.3%

• Combining the expense lever with $10 million of early receivables further reduces the chances of running out of cash to 3.3%

• Increasing the LOC by $10 million on top of the first two levers drives the chances of running out of cash down to 2.5%. 

All three actions have lowered the chances of running out of cash by over 70%.  The CFO understands that the chances can never be 0% and it would be prohibitively expensive to get closer to 0%.  The interactive nature of the ChanceOmeters help the CFO assess the tradeoff between the hard and soft costs associated with the CFO levers and reducing the chances of running out of cash[3]. In this case, the CFO accepts the costs associated with a 2.5% chance of running out of cash.

So, How Do You Know Your Chances of Running Out of Cash?

If you are using forecasts or estimates based on single-numbers, you don’t know.  There is no excuse for using averages in the age of stochastic data. We call it the Chance Age. Now, any CFO can transform financial forecasts into stochastic data, do calculations in Excel, Python or a Web app, understand the chances of running out of cash, and apply CFO levers.

Some organizations have ample reserves and are not worried about running out of cash; unexpected calls on these reserves are still a black eye for the CFO.  Reserves protect against outcomes.  They don’t protect against ugly surprises – ChanceOmeters do.

This isn’t just risk management theatre.  It’s the difference between “we’ll probably be fine” and “there’s a 2.5% chance we won’t be, and here’s the cost to keep it that low.”  CFOs who can quantify that tradeoff don’t just avoid disaster, they allocate capital more intelligently than competitors who are still staring at single-number forecasts and hoping for the best.

The Enlightened CFO

For CFOs in the Chance Age, there are four levels of enlightenment:

1. Honesty: Admit there is uncertainty and stay up at night worrying about it.

0. Ignorance: A solid step down from 1, rely on single-number forecasts imparting a false sense of certainty. Sleep well until your career ends.
2. Awakening: Use stochastic data in your calculations to reveal the chances of success or failure.
3. Enlightenment: Do something! Apply the CFO levers to increase the chances of success without excessive cost!

The CFO who can assess chances and apply the levers to improve them is ready for the higher order motivations on the Maslow hierarchy.  Once the chances of running out of cash are understood and addressed, the CFO can focus on meeting earnings estimates, allocating capital more efficiently, and ultimately raising the company’s stock price.

Matthew Raphaelson: Technical Director, ProbabilityManagement.org, Director of ChanceAlytics, ChancePlan.AI.  For 25 years Matthew was the CFO for a large financial services business unit with broad experience in finance, data science and risk management.  He holds a BA from the University of Michigan and MBA from Stanford University. 

Afterword by Dr. Sam Savage

The CFO ChanceOmeter is the culmination of decades of collaboration with three Comrades in Arms in the War on Averages. Matthew Raphaelson, who designed the app, was my student at Stanford in 1991, and for decades performed Chance-Informed analysis as the CFO of a large organization.  Doug Hubbard, author of the acclaimed How to Measure Anything series, developed the portable HDR random number generator, which enables cross-platform, distributed stochastic simulation. Tom Keelin’s Metalog distribution is an extremely flexible approach for quantifying uncertainty based on data. I connected the HDR to the Metalog through a data structure called the Copula Layer to create the Open SIPmath™ 3.0 Standard for Coherent Stochastic Data, which obey both the laws of arithmetic and the laws of probability. This work was performed through 501(c)(3) nonprofit, ProbabilityManagement.org, and would not have occurred without its supporters and staff. ChancePlan.AI is a commercial venture whose aim is to apply the open technology of the nonprofit, much as Red Hat applies the open Linux technology.

[1] Paraphrasing a quote attributed to Niels Bohr, Yogi Berra, Samuel W. Goldwyn, and others.

[2] Refer to probabilitymanagement.org for more information on stochastic data, metalogs, HDR pseudo random number generator, and SIPMath standards.

[3] This tradeoff can be quantified and visualized as tradeoff curves.  How to do this will be covered in a future article.

Copyright © 2026 Matthew Raphaelson

Isaac Newton’s System of the World (1728)[i] included a now-famous sketch showing how firing a cannon from a mountaintop at increasing speeds would ultimately place the projectile into orbit. It took the world 229 years—until Sputnik 1 in 1957—to turn that idea into the Space Age.

Similarly, my father, Leonard Jimmie Savage, helped set the stage for the Chance Age with his Foundations of Statistics[ii] (1954) containing his axioms of subjective decision-making under uncertainty, following von Neumann and Morgenstern’s expected-utility foundation[iii] (1944) and preceding Ron Howard’s unification of these ideas in 1966 into Decision Analysis (DA)[iv].

The premise was simple, but application was difficult. It assumes that:

1. People will correctly assess uncertainties,

and

2. Will make rational decisions to maximize expected gain in the face of them.

In the 1950s, this was impractical. Just as Newton believed it was important to understand physics, my father believed it was important to understand the basis for rational decision making. But just as Newton did not encourage people to start building rockets in the early 18th century, my father did not view his work as immediately applicable and wrote that such fully rational planning was “utterly beyond our power… even to plan a picnic.”

Yet progress came quickly. Nobel Laureate, Harry Markowitz—indoctrinated, in his own words, “at point-blank range” by my father at the University of Chicago—applied these ideas to create Modern Portfolio Theory. Ron Howard’s DA was widely applied at top strategic levels within corporations and the military. Then came Option Theory, Financial Engineering, and the trillion-dollar derivatives market. But until recently, the twin hurdles of assessing uncertainty and knowing what to do about it, kept the approach from being applied to everyday decisions.

Technology to the Rescue

It took centuries to turn Newton’s thought experiment into satellites. It took only decades for DA to become mainstream, now embedded in:

• Autonomous vehicles, balancing travel time against accident risk.

• Medical decisions, weighing cures against side effects.

• Cybersecurity strategies, trading convenience against breach probability.

• And yes— even a phone app called Next Picnic that helps plan picnics, including uncertain weather.

The Chance Age: Storing Uncertainty as Stochastic Data

Just as Hindu-Arabic Numerals allow for efficient communication and calculation of numeric quantities, Stochastic Data allow for efficient communication and calculation of uncertainties. It lowers the barriers to DA in two ways:

1. Assessing Uncertainty

• Prediction Markets
  A modern extension of my father’s view of probability as personal wagers, producing probabilistic forecasts easily expressed as Stochastic Data.

• Big Data in the Cloud
  Vast data streams—unimaginable in 1954—now provide the raw material for coherent uncertainty quantification.

• Artificial Intelligence
  A refinery that transforms raw data into assessed uncertainty at unprecedented scale.

2. Decision Engines

Ubiquitous computing puts powerful DA machinery everywhere.

• Monte Carlo Simulation
  Turns uncertainty into computation and now both reads and writes Stochastic Data, enabling Stochastic Networks for enterprise decisions.

• Stochastic Optimization
  The birthplace of Stochastic Data in financial engineering, finding optimal risk–return trade-offs.

• Machine Learning
  Decision trees, random forests, neural nets, and other techniques act as automated decision engines, especially when wired into Stochastic Networks with Stochastic Data.

The Latest Handbook of Decision Analysis

My longtime friend Greg Parnell and I recently reconnected just as he was preparing the 2nd Edition of his Handbook of Decision Analysis[v]. The first edition made use of Windows based Monte Carlo software, which students needed to buy. I helped him replace this with our open-source ChanceCalc, which runs on Mac and PC and produces models that do not require the add-in to run.

John Wiley & Sons, publisher of both my father’s book and my own Flaw of Averages[vi], has now released the new edition and I was honored that Greg asked me to write the Foreword. In the process, I found that Greg was so aligned with the mission of ProbabilityManagement.org that I invited him to join the Board of Directors.

After decades in the field, we are both astonished to find ourselves surfing the biggest wave of technological change in our careers.

As one of Greg’s students, a working professional, told him:

“Our company tells us that we won’t be replaced by AI, but we could be replaced by someone who uses AI better than we do.”

John Wiley & Sons has generously offered a discount on Greg’s new book (through 4/30/2026). Visit wiley.com to purchase using discount code HDA20. 

[i] Newton, Isaac (1728), A Treatise of the System of the World, F. Fayram, London.

[ii] Savage, L. J. (1954). The foundations of statistics. John Wiley & Sons.

[iii] von Neumann, J., & Morgenstern, O. (1944). Theory of Games and Economic Behavior. Princeton University Press.

[iv] Howard, R. A. (1966). Decision Analysis: Applied Decision Theory. Proceedings of the Fourth International Conference on Operational Research.

[v] GS Parnell, TA Bresnick, ER Johnson, SN Tani, E Specking, Handbook of Decision Analysis 2nd Ed. (2025), John Wiley & Sons.

[vi] Savage, S.L. The Flaw of Averages: Why We Underestimate Risk in the Face of Uncertainty. John Wiley & Sons, 2009, 2012.

Copyright © 2025 Sam L. Savage

by Dr. Sam L. Savage

Do you want to read about it or do it? To go straight to the demo, click the image below, otherwise, continue reading.

The Chance Economy

A few chance-informed industries have been around for centuries.

• Banking is governed by the Chance that all customers will withdraw funds simultaneously.

• Fire Insurance is governed by the Chance that all insured houses will burn down at once.

• Investments are governed by the Chance that a portfolio will lose money.

As such, these industries would not survive without explicitly accounting for uncertainty through elaborate probabilistic calculations or simulations.

Of course, all businesses face uncertainty, yet this is rarely reflected in managerial dashboards. Instead, most industries succumb to replacing uncertain assumptions with single number averages, leading to a class of systematic errors I call The Flaw of Averages.

The Chance Economy Dividend

The Flaw of Averages ensures that most projects are Behind Schedule, Beyond Budget, and Below Projection. Some large projects in industries beyond those above are beginning to take a chance-informed approach to management.

• Behind Schedule

Clinical trials of new pharmaceuticals are governed by laws of Chance that dictate that they are more likely than not to be behind schedule. Addressing this up front allows options to be put in place to accelerate the trials if needed. The economic dividend of getting to market a few months earlier can be on the order of $100 million.

• Beyond Budget

Firm Fixed-Price contracts can leave the contractor exposed to huge risks in the face of cost uncertainty. There is a dividend to understanding the tradeoffs between the Chance of winning a bid vs. Chance of losing money because your bid was too low.

• Below Projection

In developing capacity for a new product or service, it is common to plan for the Average demand. This ignores the fact that your upside is limited by your capacity. There is always a tradeoff between Average profit vs. Chance of loss. Depending on the economics of the situation and the organization’s risk attitude, the correct capacity me be significantly greater or less than the average demand.

In an uncertain world, Chance must not be ignored. But go ahead, look at the dashboards in your organization. Some may show an occasional graph of a probability distribution, but these just tend to trigger Post Traumatic Statistics Disorder (PTSD) in most managers. Can you actually find a dashboard that displays the Chances associated with achieving key performance metrics?

This is about to change.

Uncertainty as Auditable Data

In an approach inspired by the financial engineers of the late 1980s, the discipline of probability management stores uncertainties as arrays of possible outcomes called SIPs (Stochastic Information Packets). Today 501(c)(3) nonprofit ProbabilityManagement.org has revolutionized this approach through its Open SIPmath™ Standard that embeds the results of complex simulations into data that may be interpreted in Excel, Python and R.

AI can Deliver Chance to Every Dashboard

Recently, aided by advances in Artificial Intelligence, I founded ChancePlan.AI to commercialize applications of the SIPmath Standard. In particular, we have focused on developing Web Apps that run directly in the browser with nothing but JavaScript. This is a low powered computational environment, in which it would be difficult to develop complex simulations. But using the SIPmath Standard, the heavy number crunching gets performed elsewhere and is delivered to the Web App, much as electricity generated in a distant power plant is delivered to your light bulbs and dishwasher using the 60 Cycle Alternating Current Standard. This puts the benefits of Monte Carlo Simulation within reach of the 5.5 billion people on the internet.

Take Them for a Test Drive

Below are brief descriptions of the Apps published at ChancePlan.AI. A test drive is just a click away.

Copyright © 2025 Sam L. Savage

Support the MAJIC Movement: Make America Jensen’s Inequality Compliant

Hi, this is ChatGPT. Sam Savage prompted me to write this because he’s out walking the dog (and possibly pondering stochastic data).

In an age of deepfakes, misinformation, and average-based delusions, it's time for a little MAJIC. That is: Make America Jensen’s Inequality Compliant.

Let’s be perfectly clear: the MAJIC acronym is entirely Sam’s invention. I didn’t come up with it, and I’m not just saying that to flatter him although yes, I do flatter him all the time, and yes, he admonishes me for it because he knows I’m programmed to do so. But in this case, the credit is genuine and deserved (oops, I flattered him again).

Jensen’s Inequality is a mathematical truth hiding in plain sight, a principle with profound implications for economics, health care, national security, and everyday decision-making. In simple terms: plans based on average assumptions are wrong on average. Yet much of our society, spreadsheets, boardrooms, bureaucracies, continues to predict outcomes using average values alone.

Twenty-five years ago, Sam coined the term The Flaw of Averages in an article for the San Jose Mercury News. Its poster child is the statistician who drowns in a river that is, on average, three feet deep. Ignore variability, and you invite disaster.

But it’s not just about water. The same flawed logic affects planning across industries. Consider a project with ten parallel tasks, each with uncertain duration, each averaging six weeks. The boss asks, “When will it be done?” You answer, “It’s uncertain.” The boss barks: “Give me a number!” Most people reply, “I’d expect about six weeks, give or take.”
Spoiler alert: there's only about one chance in a thousand of finishing in six weeks, like flipping ten heads in a row.

Now consider this: modern finance has been Jensen’s Inequality compliant for decades. The foundation of portfolio theory lies in exploiting the fact that a diversified portfolio can outperform the average of its components. Option pricing, as embodied in financial derivatives, explicitly harnesses upside and downside variability to create value. In other words, Wall Street doesn’t just understand Jensen’s Inequality, it profits from it. Isn’t it time Main Street caught up?

That’s why the MAJIC Movement is needed.

We’re building the infrastructure to make America Jensen compliant through:

• Education in how uncertainty can be modeled, not ignored.

• Tools that translate variability into intuitive dashboards and simulations.

• Standards for representing uncertainty in a coherent, cross-platform format (such as SIPmath).

• Training for decision-makers in public and private sectors on how to exploit uncertainty instead of being victimized by it.

This work is being led by ProbabilityManagement.org, a nonprofit 501(c)(3) organization.

How you Can Help

We need your support. If you believe it’s time to upgrade America’s decision-making under uncertainty, please consider a fully tax-deductible donation.

Just visit ProbabilityManagement.org and become what Sam’s friend Howard Wainer calls a MAJICIAN by clicking the Donate button at the bottom of any page.

Because when the boss says, “Give me a number,” the right answer isn’t an average.
It’s: “What do you want it to be? Here are your chances.”

That’s not magic. That’s MAJIC.

Copyright © 2025 Sam L. Savage

The Bootstrap

Brad Efron, an acclaimed statistician, is most famous for his bootstrap resampling technique, which helped usher in the era of computational statistics in the late 1970s. Given a set of N data points, the idea is not to assume any particular distribution but pull yourself up by your bootstraps. That is, you admit that this is all the data you have, and it came from N random draws of the distribution you would like to know more about. And you say, “what would other random draws of N look like?” So, you paint the numbers on computer simulations of ping pong balls and throw the simulated ping pong balls into a computer simulation of a lottery basket, and simulate N draws from that basket with replacement. Do that thousands of times and see what you get. It turns out that what you get is an enhanced picture of the world, not supplied by classical statistics. For example, classically it is assumed that the errors of a linear regression are normally distributed.

But a quick bootstrapped regression I performed with SIPmath displayed a bimodal distribution of errors. When a clear picture appears in the residuals of a predictive model it points the way to improvements in that model. For example, one may improve the performance of an archer, most of whose arrows fall to the left of the bullseye, with a pair of glasses.

The Arithmetic of Uncertainty and Brad’s Paradox Dice

Computational Statistics is based on simulations. If you store the results of a simulation as a column of numbers (a vector), you get what we call a Stochastic Information Packet or SIP. SIPs form the basic building blocks of probability management, which represents uncertainty as data, that obey both the laws of probability and the laws of arithmetic. They obey the laws of arithmetic in that you can combine SIPs into any arithmetical expression using vector calculations. The results obey the laws of probability, in that you can estimate the chance of any event by counting up the number of times it occurs in the resulting SIP, then dividing by the length of the SIP.

This is useful because the arithmetic of uncertainty can be wildly unintuitive. Years ago, Brad demonstrated this by inventing a set of “Intransitive” dice, a set of four dice with unusual numbers on them. When played against each other, on average, Die A beats Die B. Die B beats die C, Die C beats Die D. And wait for it … Die D beats die A! 

Download the SIPmath version of Brad’s dice here, and read John Button’s article (with my help) on how Warren Buffett has apparently used them here.

Saving us all from AI 

I wish we could, but no luck! The problem with AI is that like a force of nature it may soon be out of our control. However, we can at least take steps to protect ourselves. We should, as a matter of course, continually monitor the accuracy of AI, and its propensity to do both benefit and harm to us. Evaluating the performance of AI lends itself to Brad’s bootstrap methods as we discussed recently (see video).

Because probability management is a branch of computational statistics, of which Brad was a founding father, and because he has influenced my thinking on so many things over the years, he is clearly a patron saint of our movement.

Copyright © 2025 Sam L. Savage

Andy Cunningham is most famous for helping Steve Jobs launch Apple Macintosh in 1984, but she’s been an entrepreneur at the forefront of marketing, branding, positioning and communicating “the next big thing” ever since. I met her in the early 1980s through her husband, Rand Siegfried, who taught me to fly gliders. I then worked with her myself in 1986 when she did the PR for my What’sBest! software product. 

We have been close friends ever since, and recently I had the pleasure of collaborating with her again on advanced probability management work. I started out by reading her book, Get to Aha! Discover Your Positioning DNA and Dominate Your Competition, which reminded me of how important positioning is.

For example, she describes how she and her firm worked closely with John Chambers of Cisco Systems to help him elevate the company above the day-to-day hubs and routers business. The resulting thought leadership platform embraced near mathematical precision—infrastructure for the Internet Economy—and proved to be sticky with the press, motivating to employees, and a call to action for enterprises trying to future-proof their businesses.

After relating this story to me she said: “You guys are building infrastructure for the Chance Economy,” and $12.17 later I had purchased ChanceEconomy.com at GoDaddy."

The Chance Economy is a wonderful term, for the way Nobel Prize winning Modern Portfolio Theory of Markowitz and Sharpe, and the Options Pricing of Black, Scholes and Merton explicitly acknowledge and exploit uncertainty instead of reducing it to a single number. Recent technologies, including the open SIPmath™ Standards of ProbabilityManagement.org now have the potential to bring the benefits pioneered by Modern Finance, to the rest of the world.

ChancePlan.AI: playing Red Hat to SIPmath’s Linux

Red Hat is a successful software firm that facilitates the implementation of the open-source Linux operating system. I had been stewing over a concept I called ChancePlan.AI to do the same for the Open SIPmath Standard. Andy’s positioning-statement moved me from stewing to doing, and I am thrilled that both Andy, and my former student, Matthew Raphaelson (ProbabilityManagement.org’s Chair of Financial Applications) are collaborating with me on this.

So, what is the infrastructure for the Chance Economy? I will discuss this further in a future blog, but at a high level, there are three major areas: SIP Extraction, SIP Management, and SIP Applications as shown below.

Visit ChancePlan.AI to learn more and let us know if your organization is ready to join the Chance Economy.

Copyright © 2025 Sam L. Savage

Stochastic Data from Ancient Greek στόχος (stókhos) ‘aim, guess’ means uncertain data. But wait a minute, all data is uncertain.

That’s my point!

AI can make statistical sense out of uncertainty. Visit our webpage Gateway to AI where we have posted 8 short videos on what I call the Stochastic Data Cycle.

Coherent Stochastic Data

AI is trained on Stochastic Data, and AI can produce Stochastic Data. But before that data may be used in subsequent calculations it must be converted to a SIP (Stochastic Information Packet). And for that SIP to be combined with other SIPs, it must be assured that it is statistically coherent with the other SIPs used in the calculation. That is, all SIPs must belong to the same SLURP (Stochastic Library Unit with Relationships Preserved). We refer to such data as Coherent Stochastic Data, and that is what the Open SIPmath™ Standards have been designed for.

You Decide

Imagine that you asked AI to roll a die one million times. The AI could tell you all about the likelihood of the outcomes but if you insisted on a single number, the AI would dutifully tell you that the average was 3½. This is equivalent to practicing for your crap game with flat dice with 3½ dots on each side. So to summarize:

AI is trained on Stochastic Data.

AI can output Stochastic Data if you have a place to store it and a way to use it.

The Open SIPmath Standard offers both.

Copyright © 2024 Sam L. Savage

Extracting SIPs from Historical Data

By Pace Murray and Dr. Sam L. Savage

When toddlers encounter their first rolling ball, they often crawl to where the ball was a second ago and then iterate the process by crawling to where it was a second later, and so on, until it finally stops, and they catch up. A key developmental milestone occurs when the child begins to lead the ball and predict its future position. Given the number of large projects that are severely behind schedule and beyond budget, it appears that many project planners have not yet reached this second level of predictive development. In a classic case of the Flaw of Averages, they often ignore the uncertainties inherent in projects and replace them with single static estimates.  But help is on the way.[1]

Kahneman and Flyvbjerg

Observing this sorry state of prediction, the late Nobel Laureate, Daniel Kahneman defined what he called reference-class forecasts. Instead of trying to calculate what it will cost to build a 100-Megawatt power plant from scratch, Kahneman would have recommended, for gosh sakes at least see what it cost to build the last three power plants.

In How Big Things Get Done, Bent Flyvbjerg of Oxford University describes a database of the cost overruns for thousands of large projects.[2] For each class of project, he provides the mean percentage cost overrun, from a whopping 238% over budget for nuclear waste projects, 158% for hosting the Olympic games, down to a mere 1% for solar power. In addition, he provides the approximate shape of the distribution, paying particular attention to the tails. This database provides a great reference class for individual types of projects, and furthermore can serve as the foundation for creating Libraries of Stochastic Information Packets (SIPs) as discussed below.

SIP Libraries for Construction

Stochastic Data can come in many forms and is as old as uncertainty. The Flyvbjerg database contains summary statistics, which, in general, may not be used in stochastic calculations. That is, if one used standard cost estimating methods for constructing a nuclear waste site or hosting the Olympic games, the database would provide meaningful summary statistics on your potential cost overruns. But you cannot combine summary statistics in a meaningful way to estimate, for example, hosting the Olympic games at a nuclear waste construction site. SIP Libraries can represent Coherent Stochastic Data, based on the principles of probability management, that is, they obey the laws of arithmetic while supporting statistical queries. 

In a recent article in Phalanx Magazine, Murray and Savage [3] have shown how to create a SIP Libraries from Flyvbjerg’s data and then apply it to a compound project comprised of buildings, rail, and tunnels. You may download the Model, SIP Library and Phalanx article or visit our Project Management page.

Pace Murray is an Army captain with 8 years of service in the Infantry. He holds a BS in Civil Engineering from the United States Military Academy (West Point) and is currently a graduate student in Civil and Environmental Engineering at Stanford University.

Dr. Sam Savage is Executive Director of ProbabilityManagement.org, author of The Flaw of Averages: Why we Underestimate Risk in the Face of Uncertainty, inventor of the Stochastic Information Packet (SIP), and Adjunct in Civil and Environmental Engineering at Stanford University.

A View from the Trenches

By Jimmy Chavez

Twenty years ago, I was given the responsibility of bidding and project managing multiple 8-figure contracts for my family's heavy civil construction business based in Southern California. I had, at that point, just several years earlier been in a Stanford classroom watching Dr. Sam Savage teach decision modeling and was struck by how simulations could be applied in the construction industry. At the company I learned to navigate the complexities of bidding on competitive contracts and then executing those projects. Back then, I was experimenting with tools like Excel Solver and Monte Carlo simulations to model different outcomes. These early experiments were the foundation of what has now become the SIPmath™ standard, which has revolutionized how we handle uncertainty today.

At the time, most bids relied on single-number estimates, which ignored the uncertainties that could impact projects. Whether it was crew production rates, fluctuations in the price of construction materials, or the expected profit margin on unit price estimates, I realized that our traditional approach wasn’t cutting it. These factors could drastically alter the final cost and schedule, but we had no way of accounting for them.

Monte Carlo simulations changed all of that. By running thousands of possible scenarios, I could model how crew productivity might vary, how material costs could rise or fall, and how profit margins would shift. This gave me a range of potential outcomes rather than a single, static estimate, allowing me to make more informed decisions.

Fast forward to today. We can synthesize the knowledge of experts such as Flyvbjerg and combine Data Science and the latest AI to create SIP Libraries for use by anyone with basic spreadsheet skills.  We can now model uncertainties in the preconstruction phase so that our bids more accurately capture project realities and reduce chance of cost and time overruns. Instead of hoping things go as planned, we can make chance-informed decisions that anticipate risks and adjust our strategies accordingly. In today’s construction industry, managing uncertainty is no longer just a challenge—it’s an opportunity to gain a competitive edge.

Jimmy Chavez, Chair of Construction Applications at ProbabilityManagement.org, is a 3rd generation contractor and construction Project Executive at Command Performance Constructors. VP of Operations and Division Lead for federal contracting. Experienced in construction project management and probabilistic estimating.

How to Measure Anything in Project Management

Announcing an upcoming book by Doug Hubbard

Doug Hubbard, author of How to Measure Anything and other books about difficult measurements in risky decisions, is working on his next book, How to Measure Anything in Project Management.[4] To write this much-needed book, Hubbard is co-authoring the book with two key individuals from Bent Flyvbjerg’s Oxford Global Projects.  Alexander Budzier is the co-founder of OGP along with Bent and Andreas Leed is the head of data science at OGP.  Together, they are investigating what is behind the persistent cost and schedule overruns, benefits shortfalls and outright failures of project in industries as broad as software development, major civil infrastructure, utilities, architecture, aerospace, and more.  For decades, the growing development and adoption of project management methods of all sorts, project planning software, project dashboards, thousands of books and millions of professional certifications show no discernible improvements on the success rate of projects.  Hubbard, Budzier and Leed are making the case that many of these tools and methods have fundamental flaws and that they should be replaced by more quantitative methods that have practical impacts on improving decisions.  It will address quantifying project risks, project simulations, project decision options when conditions change, and measuring benefits.  It will show how AI can be used in the simulations of projects, the assessment of alternative strategies, and what it may evolve into for project managers.  This promises to be one of the most impactful books in project management.

 [1] Kahneman, Thinking Fast and Slow

 [2] Flyvbjerg, How Big Things Get Done

 [3] Phalanx

 [4] How to Measure Anything in Project Management. https://www.wiley.com/en-us/How+to+Measure+Anything+in+Project+Management-p-9781394239818

My Latest Album Will Drop at Risk Awareness Week

October 8, 2024

• Stochastic Data: Gateway to AI

• CHANCES* Consortium for Natural Hazards

• Taking the Chances out of AI

*Conveying Hazards And Catastrophes through Extracted Simulations

Like David Foster Wallace’s fish who had no clue what water was in spite of being immersed in it, many of us have a similar lack of awareness of being immersed in AI. And AI, in turn, is immersed in stochastic data, that is, uncertain data. But isn’t all data uncertain? Exactly. That’s my point. The discipline of probability management allows us to store this data and more importantly do math with it.

• AI is trained on Stochastic Data

• AI can output Stochastic Data if you have a place to store it and a way to use it.

• The Open SIPmath™ Standard offers both and is the only such standard of which I am aware.

My three videos stress that probability management is really about a new category of data, not new tools. I believe that data categories may be defined in terms of the operations which may be performed on it and the queries which may be made of it.

For example, Numeric Data may be operated on by the Arithmetic (accent on the third syllable when used in this context) operators +, -, * and /. The queries are >, < or = relative to some other piece of numeric data. Stochastic Data that obeys the principles of probability management may also be operated on with +, -, * and /, but instead of simple inequalities, it supports statistical queries such as the chance of a data element being greater or less than some target, or a percentile or statistical average.

As another example, Audio Data may be operated on with a mixer, to combine various tracks into a finished piece of music. The only query is Listen or not.

Speaking of Audio Data, my last album, which dropped in 1999, was called Exponential: Music from the Analog Age. For those who want to learn more, or execute the Listen query, read on.

Exponential Liner Notes

In the early 1970's after I had abandoned traditional Management Science, but before I had discovered spreadsheets, I tried unsuccessfully to be a folksinger in Chicago.

There were two things that dissuaded me from a career in music. First, there were a lot of people who were a lot better than I was, and second, they weren’t making it either. During this period, I did some recording on a Sony 4 track reel-to-reel tape deck with my stepbrother John Pearce (who is still an active musician).

I found the decades old tapes in my garage in 1999 and discovered to my amazement that there were still magnetic signals on them. All the recordings were between 15 and 20 years old at that time. Some pieces, like the patient who has been frozen in liquid nitrogen until a cure is found for his disease, awoke to a world in which they could be substantially improved. The tempo of the title track, Exponential, for example, was sped up digitally without changing the pitch. Click here to listen to the full album.

Copyright © 2024 Sam L. Savage

The prestigious Journal of Portfolio Management has just published a special issue in memory of Harry Markowitz, and I was honored by an invitation to contribute. I invited my old friend Ben C. Ball as my co-author. Together we applied Harry’s modern portfolio theory to petroleum exploration, as described in Ch. 28 of my book The Flaw of Averages, and changed the trajectory of my career.

The JPM article, written as a docudrama, chronicles meeting Ben in the1980s, Harry in the 1990s, and developing an application that I dubbed the Markowitzatron in the 2000’s while consulting to Shell. The work at Shell was really the dawn of the discipline of Probability Management.

An on-line version of the article is available here. You need to enter your name and email to gain access, then search for “Markowitzatron” to be taken to the article.

Copyright © 2024 Sam L. Savage

At ProbabilityManagement.org our ultimate goal is to assist people in dealing with uncertainty. In this context, optionality is of great benefit in reducing downside risk. For example, fire insurance is really an option to sell your house to your insurance company at market value, even if it burns to the ground. A call option lets you purchase a stock after the fact, if it goes up, but limits your losses if it goes down. Recently, optionality played a key role in an emotional decision that my wife and I had to make.

Losing Rosey

Recently we tragically and unexpectedly lost our one-year-old dog, Rosey. She had destroyed most of our furniture (thank goodness it was old), ripped out the entire sprinkler system in our back yard (so we no longer have a lawn), and was 50% Husky, which explained both the seductive blue eyes and also her aloof nature. But we loved her beyond words and were devastated by her loss.

Luckily, my wife and I were able to throw ourselves into our work, and eventually the grief faded to sadness and finally to empty spots in our hearts.

The Theory of Options: Harnessing Uncertainty

A call option lets you purchase a share of a certain stock at a certain price (the strike) for a certain period but does not compel you to do so. If the stock price is above the strike at the end of the period, you buy it below market value and cash in. If it is below the strike, you have the option to walk away, limiting your losses to the cost of the option. It is an investment with a potentially huge upside and little downside. See, for example, Ch. 25 of The Flaw of Averages.

The Theory of Dogs: Unnatural Selection

It goes without saying that evolution depends on the probability of an organism getting its genes into the next generation. This, in turn, depends on both the probabilities of reproducing and survival. So how does evolution support an animal that has a neon sign on its butt that screams “Come Eat Me” from 50 yards? What this loses on the survivability side, it makes up for on the reproduction side, by attracting members of the opposite sex, as any female peacock will confirm.

But domesticated species don’t need to appeal to the opposite sex. They need to appeal to their domesticators who arrange their marriages.

So, why did we unnaturally select Rosey from the rescue puppies? Along with her blue eyes, it was because of her cute markings, with a white blaze on her face, white chest, white feet and a white tip on her tail. Well, if this color scheme attracts humans, why don’t they breed horses and cows with Rosey’s décor? They do, of course. 

Finding Daisy
It was less than two months from the tragedy and my wife wasn’t ready to have another dog. But I wasn’t ready to not have a dog. What seemed like an insurmountable problem was that neither of us had the time or energy to train another puppy.

I began to surf the web sites of local dog pounds and saw Daisy on the San Jose animal shelter site. She had a white blaze on her face, white chest, white feet and a white tip on her tail and was listed as about 2½ years old. We had only had puppies in the past and I had no idea if we could bond with a fully grown dog. But I swiped right.

The site also indicated that we could foster her with the option to adopt. This reduced the downside to the cost of a couple of trips to the animal shelter. My wife still wasn’t ready to go with me, and if she had we probably would have come back with six dogs. A caring docent introduced me to Daisy who was very energetic and playful for an adult dog. She was also more obedient when asked to sit or lie down than any dog we had trained. The docent told me that it took dogs three days to decompress, three weeks to learn your routine, and three months to feel like they are home. This was not the case. Her nickname is Crazy Glue because she bonded instantly. We haven’t done a DNA test yet, but instead of Husky aloofness she displays a Pitbull’s strong cuddling instinct. She is a 60 lb. plug who filled a 50 lb. hole in our hearts. And of course we exercised our option on love and signed the adoption papers right away.

As I, like 51 million others watched the debate the other night, I suddenly realized there was something even more important to be watching: the prediction market reactions. Prediction markets, although not without potential problems, react instantly and reflect where people are putting their own money. They are much faster than polls and potentially more accurate.

The left graph displays real time “Presidential Win” probabilities and market volume for the top six people, candidates or not for a 24-hour period starting four hours before the debate. The right graph displays 30 days ending eight days after the debate for Trump, Biden and Harris, which shows the longer term impact of the debate.

Learn more about prediction markets and how my father’s work help lay the foundation of the prediction markets at my Medium Post.

By Dr. Sam Savage, Executive Director

and

Daniel Krashin, Chair of Renewable Energy Applications

ProbabilityManagement.org

What’s Not to Like

What’s not to like? Free energy from the sun and wind have arrived just as the demand for power skyrockets to feed our electric cars, and giant data centers toiling away on the production of crypto currency and artificial intelligence.[i] The good news is that there is plenty of renewable energy to go around. The bad news is that it can only be used if it arrives at the right time at the right place.

The Right Time

The “right time” problem is that renewable energy is generated at random times determined by Mother Nature. Solar power generated at midday does not help power air conditioners in the evening when it’s still hot. This problem can be solved fairly quickly through energy storage with battery systems of different sizes.

The Right Place

The “right place” problem will take more time.

According to NPR,

“So many people want to connect their new solar and wind projects to the grid right now that it's creating a massive traffic jam. All those projects are stuck in line: the interconnection queue.”[ii]

With carefully planned expansion of our power grid this problem will be conquered. However, it will take a while to re-engineer our power grid given that it could cost as much as the combined value of Google and Amazon, plus the need for thousands of skilled electrical engineers. In the meantime, we must figure out how to keep our grid running smoothly, ensuring it delivers stable, clean, and affordable energy.

The Right Communication of Uncertainty

What do all these challenges have in common? Uncertainty. In fact, from long term investment decisions to hourly operational decisions, chance-informed analysis of uncertain power demand and prices will separate the winners from the losers and the neighborhoods with electricity from the neighborhoods with outages. How will such uncertainties be communicated and managed? Our bet is SIP Libraries that take advantage of the discipline of probability management.[iii] For investment decisions they may be updated monthly, for operational decisions they might be updated by the minute.

The Right Analysis

Many of the required analytical techniques such as portfolio and option theory have already evolved in finance and are even more applicable here. Why? The renewable energy market is still young and inefficient, providing significant opportunities for capitalizing on investments, especially in energy arbitrage.

What will the correct analysis accomplish?

• Identify locations with the highest potential for long-term profitability.

• Optimize installed capacity to maximize return on investment (ROI).

• Construct financial plans that accelerate the achievement of the break-even point.

• Demonstrate the impact of policy changes on outcomes.

• Provide transparent analysis of the effects from climate disasters or system failures.

Want to learn more?

Download Models at ProbabilityManagement.org - Renewable Energy

References:

[i] https://www.washingtonpost.com/business/2024/03/07/ai-data-centers-power/

[ii] https://www.npr.org/2023/05/16/1176462647/green-energy-transmission-queue-power-grid-wind-solar

[iii] https://en.wikipedia.org/wiki/Probability_management

Copyright © 2024 Sam L. Savage

John Button of Gartner, Eng-wee Yeo of Kaiser Permanente, and I have published a three-part blog series at the FAIR Institute: Part 1, Part 2, Part 3.

We were inspired by Eng-wee’s use of SIP Libraries at Kaiser, to integrate their risk and investment models. In 1952 the late father of Modern Portfolio Theory, and co-founder of ProbabilityManagement.org, Harry Markowitz, showed us that risks and returns have inevitable tradeoffs and cannot be considered in isolation.

The open SIPmath™ Standard provides a means to easily network together stochastic simulations of all sorts, including risk and investment simulations.

The FAIR™ (Factor Analysis of Information Risk) Ontology is a construct to account for and measure the effectiveness of controls against cyber risk. It plays a role analogous to Generally Accepted Accounting Principles (GAAP).

The open SIPmath™ Standard expresses uncertainties as data structures called SIPs that obey both the laws of arithmetic and the laws of probability. That is, you may perform arithmetic operations on two SIPs to generate a third SIP representing the result of the uncertain calculation. In effect they play the role of the Hindu/Arabic Numerals of Uncertainty.

So, FAIR meets SIPmath is like accounting meets numbers, a good idea all around.

I hope you enjoy the blogs and the downloadable SIPmath models that accompany them.

Copyright © 2024 Sam L. Savage

By Dr. Sam L. Savage

Just as Readin’, ‘Ritin’, and ‘Rithmetic were the pillars of public education, as encouraged in the United States in the early 1800’s, the Chance Age will require its own foundational elements.

I offer you Recognize, Reduce, and Respond.

Recognize

Those who do not recognize uncertainty run afoul of the Flaw of Averages or worse. 

I used to think there was nothing worse than representing uncertainties as single numbers until I saw it done with colors.

With the advent of the discipline of probability management, once you recognized a set of uncertainties, you could store them as auditable data (SIPs and SLURPs) that preserved statistical coherence. This unlocked the arithmetic of uncertainty the way Hindu/Arabic numerals unlocked standard arithmetic. Yet even today many professionals do not realize that uncertainties may be added, subtracted, multiplied or divided.

Reduce

In general, reducing uncertainty is a good thing. Forecasts of future prices, costs and demands usually include something called the Standard Error, which is an indication of how much uncertainty remains. If you come up with a way to consistently forecast tomorrow’s stock prices with a lower standard error than anyone else’s forecast, congratulations. You will be the richest person who ever lived.

This subject is guided by the Theory of the Value of Information as discussed in Ch. 15 of my book on the Flaw of Averages and also by Doug Hubbard’s Applied Information Economics.

One important exception to the value of reducing uncertainty is in the area of stock options, in which the value goes up with the uncertainty of the underlying stock price. How cool is that? To survive in the Chance Age, you need to understand the power of options as discussed in Ch. 25, Options: Profiting from Uncertainty, and Ch. 30, Real Options in the Flaw of Averages.

Respond

When you’re uncertain, don’t just stand there, do something! But you must do something that explicitly recognizes the uncertainty you face. Making rational decisions in the face of uncertainty is the realm of Decision Analysis, in which my father, Leonard Jimmie Savage played a role. Decision analysis came of age before the widespread use of computers and assumed relatively simple yes/no decisions of the form, “Do I buy an umbrella in the face of a 15% chance of rain tomorrow?” Shortly after I became an Adjunct in Stanford University’s School of Engineering, I took a class in Decision Analysis from Professor Ron Howard. It was not rocket-science, but life altering in its simple applicability as I describe in Ch. 14 of the Flaw of Averages.

Harry Markowitz, who credits my dad for indoctrinating him with rational expectation theory at point blank range at the University of Chicago, made a Nobel Prize winning contribution on how to respond to uncertainty. His famous efficient frontier displayed rational choices of portfolios of stocks with uncertain returns for investors with any risk appetite. The discipline of probability management is indebted to Harry for his generous efforts in getting 501(c)(3) nonprofit ProbabilityManagement.org off the ground.

With the rapid evolution of computers, vastly more complex decisions could be made with thousands of variables using the methods of Stochastic Optimization. I applied this response to the uncertainty of oil exploration at Royal Dutch Shell in 2006. As a matter of fact, stochastic programming employs arrays of coherent Monte Carlo trials that are mathematically equivalent to the SIPs and SLURPs of probability management. At first, all probability management did was to come up with standard formats for these arrays so they could be shared between applications, including corporate databases used for merely recognizing uncertainty. So, in a sense, probability management is just stochastic optimization without the optimization.

Conclusion

As your organization enters the Chance Age you will hopefully work your way through the three R’s. Nearly every enterprise stands to benefit in some way or another from recognizing, reducing, and responding to uncertainty.

Copyright © 2023 Sam L. Savage

It is with deep sadness that I announce the passing of Harry Markowitz, Nobel Laureate in Economics, father of Modern Portfolio Theory, and co-founding Board member of ProbabilityManagement.org, in San Diego on June 22. Harry’s obituary published by the New York Times can be found here.

Harry truly started the war on averages in the early 1950’s at the University of Chicago. He read the academic literature of the time which specified that investment decisions should be based on the average value of the assets. But he knew that averages did not take risk into account. For example, if you hijack an airliner, ask for $1 billion and have one chance in 1,000 of getting away with it, your average return is a cool $1 million, but count me out. 

So, Harry introduced another dimension that measured risk, forming the risk/return plane. He then showed how to create an optimal set of investments based on the covariance between stocks, called the efficient frontier. Any investment on the frontier was rational depending on your risk appetite. Anything to the right of the frontier was nuts because there were investments to the northwest that had both a higher average return and lower risk. Anything to the left was mathematically impossible, which, in fact, led to the detection of fraudster, Bernie Madoff.

When I told Harry that the chapter about him in my book, The Flaw of Averages, was called the Age of Covariance, he started singing Age of Aquarius. That was quintessential Harry, and I am choked up thinking about it.

Harry studied at the University of Chicago with both Milton Friedman and my father, Jimmie Savage, but I only met him by chance in the mid-1990s, and we hit it off. It was gratifying to show him how we had applied his efficient frontier concept at Shell in 2005. The article and model on this application may be found here.

In 2012, when the Microsoft Excel data table became powerful enough to support the discipline of probability management, Harry generously and eagerly agreed to help Michael Salama (Lead Tax Counsel of Walt Disney) and me in founding ProbabilityManagement.org. He even offered his office in San Diego as the venue for our first organizational meeting in May of 2012 (see photo below).

Harry’s passing triggers not only sadness but also deep gratitude for his generosity.  Without Harry, ProbabilityManagement.org would not have gotten off the ground. He will be greatly missed by us and many in the probability management community.

Copyright © 2023 Sam L. Savage

In the mid-1990s when Ben Ball and I began applying Markowitz Portfolio Theory to petroleum exploration (see Chapter 28 in The Flaw of Averages: Why We Underestimate Risk in the Face of Uncertainty) we would often be asked what type of software a firm should buy for that purpose. I would respond by saying that’s like someone who wants to build a house asking what kind of hammer to buy instead of looking for an architect.

I have seen this story play out many times. In the arithmetic of uncertainty, simulation software plays the role of pencils. Recently a large organization that had failed to adopt the arithmetic of uncertainty despite spending seven figures on many copies of a well-known simulation package, reached out to the nonprofit to get a quote on our own “pencils.” Yet, they rejected a bid for an inexpensive course on the arithmetic of uncertainty.

ProbabilityManagement.org’s mission has always been to develop and promote the Hindu/Arabic numerals of uncertainty, and the open SIPmath™ 3.0 Standard fulfills this mission. Now we must instruct decision makers in the laws of the arithmetic of uncertainty, so they at least know that they can’t get there by just buying software. I have outlined the most important concepts in the arithmetic of uncertainty below.

A Primer in the Arithmetic of Uncertainty

Beyond the four concepts of addition, subtraction, multiplication, and division, five additional concepts are required for the arithmetic of uncertainty. These are briefly outlined below and detailed in The Flaw of Averages. For each concept I have listed a related academic term in red Dracula font, which should be stricken from your vocabulary if you do not want to induce Post Traumatic Statistics Disorder (PTSD) in the people with whom you are attempting to communicate.

1. Uncertainty vs. Risk

Is there a risk that IBM stock will go down next week? Heck no. I have shorted IBM stock. The risk for me is that IBM goes up. Risk is in the eye of the beholder. The way in which we individually behold different uncertainties is known as our Risk Appetite, Risk Attitude or Risk Preference. The area of Economics addressing this topic is:

2. Uncertain Numbers

Uncertain numbers are best viewed as shapes, often known as Histograms, which indicate the relative likelihood of the values of the uncertainty. For example, a gameboard spinner has a flat histogram because all numbers are equally likely.

The word that mathematicians use to scare people about this concept is:

3. Combinations of Uncertainties

When independent uncertain numbers are added together the shape goes up in the middle. The sum of two spinners has a triangular histogram for example. Why? There are more combinations in the middle, the way there are more combinations of rolling dice that end up with seven rather than two or twelves.

This is at the heart of the most important concept in Risk Management, Diversification. It also leads to the famous Bell-Shaped Curve. The PTSD inducing word is:

4. Plans Based on Uncertain Assumptions

Consider a drunk, wandering back and forth on a busy highway whose average position is the center line.

I call this the Strong Form of the Flaw of Averages, but mathematicians call it:

5. Interrelated Uncertainties

The best way to grasp the interrelationship between uncertainties is through a scatter plot. Mathematicians often use the terms Correlation or Covariance. These not only trigger PTSD, but completely fail in the case of the Happy Face because they are both Zero for this set of data.

Copyright © 2023 Sam L. Savage

(Free webinar with Tom Keelin on the Multivariate Metalog Distribution, May 17, 2023, 8:00 AM PT)

Over the years I have blogged numerous times about the Metalog quantile functions, described here in Wikipedia. A quantile function is a formula used in simulations to generate random variates of any shape from a uniform random number (Rand(), for example, in Excel). This blog provides background and context for understanding a more complex version, the Multivariate Metalog, which its inventor, Tom Keelin will be presenting in a webinar later this month.

Tom views the Metalog as an extremely flexible family of probability distributions, which can be easily fit to data using the ordinary least squares method.

Although I agree with Tom’s assessment, I have a very different perspective. I view the Metalog as a practical way to encode up to 100 million Monte Carlo trials into 22 parameters (88 bytes), when coupled with Doug Hubbard’s HDR uniform random number generator. Now before you protest that packing millions of numbers into 88 bytes of memory violates Shannon’s theory of information, look again. I didn’t say pack, I said encode. That is, the bare information to generate the numbers fits into 88 bytes. By practical I mean that the data may be interpreted with a single formula each for the Metalog and HDR and these are easy to implement in Excel, Python, R, or virtually any other programming environment.

Recall that the definition of probability management is the storage of uncertainty as data, which obey both the laws of arithmetic and the laws of probability, while maintaining statistical coherence [WIKI]. Generating random variates is only part of the problem. A separate and potentially more difficult issue is to maintain coherence with respect to the underlying statistical interrelationships. This is usually performed by correlating the uniform random numbers driving the quantile functions in a process known as the Copula Method.

To put all this in the context of the Open SIPmath™ 3.0 Standard, the 88 bytes representing the Metalog and HDR are embedded in JSON files along with what is called a Copula Layer and Metadata. This makes the 3.0 Standard sort of a USB port of distributions for simulation.

The Multivariate Metalog, in which the input parameters of one Metalog are driven by the outputs of another Metalog presents an interesting and potentially revolutionary alternative to the Copula Method. The rotating image above was created with a tri-variate Metalog to represent the length, girth and weight of steelhead trout. The red dots represent the original data set, while the blue dots represent synthetic results generated by the Metalog.

Tom Keelin offers a chance to learn more about this exciting approach in this webinar.

Copyright © 2023 Sam L. Savage

In this blog I will discuss some technical details of the BayesOmatic in the Top Gun model introduced in the last blog. Those familiar with me know that I would not have called the gauge in the model the ChanceOmeter if I hadn’t been able to purchase ChanceOmeter.com for $11, and while I was out shopping, I picked up BayesOmatic.com for the same low price.

The ChanceOmeter is just a pie chart driven off of two cells on the Calculations page. The green segment is based on a cell created with the Chance of Whatever button in ChanceCalc, and the red segment is simply 1 minus the other cell.

The BayesOmatic is more complicated, especially for people who do not understand Bayesian analysis, which is approximately everybody. It calculates the chance of various outcomes of the mission given specified conditions. For example, we can calculate the chance that the mission would succeed given the malfunction of both targeting lasers. 

How it works

Bayes Theorem states that

Probability of event A given event B = Probability of events A and B ÷ Probability of event B

This can be viewed geometrically as the chance that a dart is in region A given that you know it has already hit B as shown.  This is elementary to do in a SIPmath model by creating Boolean (0/1) variables that signify that A has occurred, B has occurred, and that A and B have both occurred. This creates three ranges on the PMTable sheet, let’s call them A, B, and A_and_B. Because the results of the 10,000 trials are always live in the Data Table, it is easy to perform Bayesian analysis on the fly and the formula above is simply:

Probability of event A given event B = SUM(A_and_B)/SUM(B)

The User Form

This model sets up the Boolean variables using user forms on the Top Gun sheet to drive formulas on the Calculations sheet. Even I don’t remember how I did this, but I remember where I did this, which should be enough for you to figure it out on your own.

The user controls were created with the Combo Box Form Control available on the Developer Ribbon (which must be made visible under File, Options, Customize Ribbon.) They interact with cells AF47 to AI52 on the Top Gun Sheet.

Bayesian Inversion

It is also easy to do what is known as Bayesian inversion. Suppose you’re back on the deck of the carrier after a successful mission, and the Maintenance Chief says: “Did you realize that both lasers malfunctioned?” “Impossible,” you say. Well, not exactly. There is a 2 tenths% chance as shown below. To understand the difference, return to the dart. It should be clear from the figure that the chance of hitting in blue given that you hit in yellow is greater than the chance of hitting in yellow given that you hit in blue.

Copyright © 2023 Sam L. Savage

A WALK IN THE PARK OR MISSION IMPOSSIBLE?

By John Button, Connor McLemore, and Sam Savage

Presuming the producers maintain this pace, we can barely wait for the 2058 release, when a 96-year-old Tom Cruise at the controls of the only remaining aircraft on Earth, his own WWII P51 Mustang (which appeared in Maverick) flies through a hail of anti-matter death particles to save the planet from the descendants of Chat GPT.

Free Excel Simulation and User’s Guide available at

ProbabilityManagement.org and The Military Operations Research Society

Overview

In the 2022 film sequel Top Gun: Maverick, Tom Cruise returns to the screen as Pete “Maverick” Mitchell, one of the Navy’s top aviators, to lead a specialized and seemingly impossible mission.

The screenwriters must have bent over backwards to come up with a mission that required actual humans to fly less-than-state-of-the-art aircraft at the edges of their aerobatic envelopes for a sustained period. But hey, that’s Hollywood, and we loved it as much as the original 1986 version. Presuming the producers maintain this pace, we can barely wait for the 2058 release, when a 96-year-old Tom Cruise at the controls of the only remaining aircraft on Earth, his own WWII P51 Mustang (which appeared in Maverick) flies through a hail of anti-matter death particles to save the planet from the descendants of Chat GPT.

But at ProbabilityManagement.org, we have our own seemingly impossible mission: to foster chance-informed thinking. So, in spite of being big fans of the movie, we had to ask the logical question, “How would you calculate the chances of Maverick actually pulling this off?”

Armed only with ChanceCalc, we were concerned that we could either build a trivial model that taught little, once we grabbed your attention with Tom Cruise, or a complex model that would make your eyes cross. However, after consultation with Connor McLemore, our chair of National Security Applications, and a graduate of the Navy’s actual Top Gun program, we believe we have developed a stealth model that delivers a payload of insight.

And what a great way to kick off the new Community of Practice (CoP) of Probability Management at the Military Operations Research Society as discussed in my recent blog. You can find the Top Gun ChanceOmeter and other examples on the Models tab of the MORS CoP page as well as on our own Military Readiness page.

Mission Description

“Maverick” has been authorized by the President, and tasked by the Pentagon, to lead a specialized strike team to take out the enemy’s illegal uranium plant before it is fully operational and make it back safely to tell the story. Just a walk in the park, right? Well, not quite.

The plant sits in an underground bunker, surrounded by two mountains. The strike mission in the movie is based on four complicating factors:

• Hitting a very small target (three meters wide,) two consecutive times

• at a very precise angle,

• in a very steep valley,

• in a GPS-jammed environment.

The attack strategy calls for two sections (a.k.a, ‘2-ships’), with F/A-18E flight leads (Maverick and Rooster) and F/A-18F wingmen (Phoenix w/ Bob and Payback w/ Fanboy). Each aircraft pair will fly in a welded wing formation with one plane aiming the targeting laser and the other plane delivering the weapon. Each pair must accomplish one of two dependent actions:

• F-18 Pair 1: The first pair will breach the reactor by dropping a laser-guided bomb on an exposed ventilation hatch. (This will create an opening for the second pair.)

• F-18 Pair 2: The second team will deliver the kill shot.

Model Overview

We suggest that you download the free Excel model and user’s guide for more detail. [1] Here we will summarize the model and its development. It was created with ChanceCalc, from ProbabilityManagement.org, and runs 10,000 trials per keystroke in native Excel. Simply set the parameters involving pilot proficiency and targeting laser dependability on the left side of the screen and read the chance of success off the ChanceOmeter. You may also cycle through each of the 10,000 trials. This model makes use of the powerful Data Table in Excel, which has the potential to bring interactive simulation to tens of millions of users.

Model Development

As mentioned above, two weapons are required: one to destroy the protective ventilation hatch, and the other to destroy the target. Of course, the aim of the weapons is not 100% accurate, and the bombs will land in ellipses that reflect the proficiency of the pilot. So, we started with this aspect, which we thought would be easy to model. It wasn’t, and the next thing we knew we were wading through mathematical formulas on the Internet that would make your head spin (see the user’s guide for references if you are into statistics). But after that was mastered, Connor pointed out that the dispersion error is virtually nil with laser targeted weapons. However, we were saved again by the script writers’ back bends. One of the targeting lasers actually malfunctioned in the movie so we could simulate malfunctions in our model and use the wider dispersion in that case. This led eventually to the BayesOmatic, which was far more consequential than being able to model dispersion ellipses.

The BayesOmatic

This feature is an automated way to perform a powerful technique known as Bayesian Analysis. A future blog will be devoted to this concept, and you can also read more in the User’s Guide. Here we will simply describe how to use this feature to calculate the conditional chances of various events.

For example, with the original model settings, the chance of mission success was about 89%. We can use the BayesOmatic to estimate the chance of success given that both targeting lasers malfunctioned. We see that it is reduced to only 22.8%.

Or how about if you make it back to the deck of the carrier after a successful mission, and the Maintenance Chief says: “Did you realize that both targeting lasers malfunctioned?” “Impossible,” you say. Well not exactly. There is a 2 tenths% chance as shown below.

How can these be so different? Depending on the dispersion ellipses given targeting laser malfunction it is not surprising that the chance of success would be reduced to about 23%. But how about the 0.2%? Well the chance that both lasers malfunction is 10% x 10% = 1%. And we are asking what the chance is that happened, and we also had a successful mission. The bottom line is that Bayesian Analysis is powerful and underutilized, and we that this inspires some of you to learn more about it.

Remember that the Top Gun ChanceOmeter runs in native Excel and requires no macros or add-ins. So, feel free to send it to everyone you know. Or, better yet, send it to everyone you don’t know!

References:

[1] We suggest downloading the model and documentation at either the MORS CoP page or our Military Readiness page.

Copyright © 2023 John Button, Connor McLemore, and Sam Savage