Deming has this pithy way of putting a lot of the problems along those lines and what he says is: “Lets play American industry and we’ll make toast. I’ll burn and you scrape.”

These links provide more details on ideas he discussed in the seminar:

• Interview of Bill Hunter, Brian Joiner and Peter Scholtes on Better Management Practices – where they discuss the improvement efforts at the City of Madison that Bill mentions in the seminar in more detail.
• Doing More with Less in the Public Sector – A Progress Report from Madison, Wisconsin
• Customer Focus: Cater to Customers Desires to Achieve Customer Delight – To serve customers, a thorough understanding of what problem you solve for customers is needed – What Could we do Better? – Stated Versus Revealed Preference

– customer focus must focus on the end users

• Keys to the Effective Use of the PDSA Improvement Cycle – how to make improvements (changes) while gathering evidence on what is working, and what changes do not result in improvement.
• In the video Bill talks about the Santayana principle (learning from history). This link is related, though from the perspective of Russell Ackoff’s ideas: Write it Down.
• Cause and effect diagram (fishbone diagram)
• Inspection is too late. The quality, good or bad, is already in the product – Cease Mass Inspection for Quality
• Ford and Managing the Supplier Relationship
• Quality In The Community One City’s Experience – Quality Comes to City Hall

Related: Design Your Organization to Serve Customers Well – Quality is a Journey to Excellence (part one)

Jeff Fry recently linked to a fantastic webcast in Controlled Experiments To Test For Bugs In Our Mental Models. I would highly recommend it to anyone without any reservations. Ron Kohavi, of Microsoft Research does a superb job of using interesting real-world examples to explain the benefits of conducting small experiments with web site content and the advantages of making data-driven decisions.

I firmly believe that the power of applied statistics-based experiments to improve products is dramatically under-appreciated by businesses (and, for that matter, business schools), as well as the software development and software testing communities. Google, Toyota, and Amazon.com come to mind as notable exceptions to this generalization; they “get it”. Most firms though still operate, to their detriment, with their heads in the sand and place too much reliance on untested guesswork, even for fundamentally important decisions that would be relatively easy to double-check, refine, and optimize through small applied statistics-based experiments that Kohavi advocates. Few people who understand how to properly conduct such experiments are as articulate and concise as Kohavi. Admittedly, I could be accused of being biased as: (a) I am the son of a prominent applied statistician who passionately promoted broader adoption of such methods by industry and (b) I am the founder of a software testing tools company that uses applied statistics-based methods and algorithms to make our tool work.

Here is a short summary of Kohavi’s presentation: Practical Guide to Controlled Experiments on the Web: Listen to Your Customers not to the HiPPO

1:00 Amazon: in 2000, Greg Linden wanted to add recommendations in shopping carts during the check out process. The “HiPPO” (meaning the Highest Paid Person’s Opinion) was against it thinking that such recommendations would confuse and/or distract people. Amazon, a company with a good culture of experimentation, decided to run a small experiment anyway, “just to get the data” – It was wildly successful and is in widespread use today at Amazon and other firms.

3:00 Dr. Footcare example: Including a coupon code above the total price to be paid had a dramatic impact on abandonment rates.

4:00 “Was this answer useful?” Dramatic differences in user response rates occur when Y/N is replaced with 5 Stars and whether an empty text box is initially shown with either (or whether it is triggered only after a user clicks to give their initial response)

6:00 Sewing machines: experimenting with a sales promotion strategy led to extremely counter-intuitive pricing choice

7:00 “We are really, really bad at understanding what is going to work with customers…”

7:30 “Data trumps intuition” {especially on novel ideas}. Get valuable data through quick, cheap experimentation. “The less the data, the stronger the opinions.”

8:00 Overall Evaluation Criteria: “OEC” What will you measure? What are you trying to optimize? (Optimizing for the “customer lifetime value”)

9:00 Analyzing data / looking under the hood is often useful to get meaningful answers as to what really happened and why

10:30 A/B tests are good; more sophisticated multi-variate testing methods are often better

12:00 Some problems: Agreeing upon Overall Evaluation Criteria is hard culturally. People will rarely agree. If there are 10 changes per page, you will need to break things down into smaller experiments.

14:00 Many people are afraid of multiple experiments [e.g., multi-variate experiments or MVE] much more than they should be.

(A/B testing can be as simple as changing a single variable and comparing what happens when it is changed, e.g., A = “web page background = Blue” / B = “web page background = Orange.” Multi-variate experiments involve changing multiple variables in each test run which means that people running the tests should be able to efficiently and effectively change the variables in order to ensure not only that each of the variables is tested but also that the each of the variables is tested in conjunction with each of the others because they might interact with one another).
My views on this: before software tools made conducting multi-variate experiments (and understanding the results of the experiments) a piece of cake, this fear had some merit; you would need to be able to understand books like this to be able to competently run and analyze such experiments. Today however, many tools, such as Google’s Website Optimizer (used for making web sites better at achieving their click through goals, etc.) and Hexawise (used to find defects with fewer test cases) build the complex Design of Experiments-based optimization algorithms into the tool’s computation engine and provide the user of the tool with a simple user interface and user experience.
In short, in 2009, you don’t need a PhD in applied statistics to conduct powerful multi-variate experiments. Everyone can quickly learn how to, and almost all companies should, use these methods to improve the effectiveness of applications, products and/or production methods. Similarly, everyone can quickly learn how to, and almost all companies should, use these methods to dramatically improve the effectiveness of their software testing processes.>

16:00 People do a very bad job at understanding natural variation and are often too quick to jump to conclusions.

17:00 eBay does A/B testing and makes the control group ~1%. Ron Kohavi, the presenter, suggests starting small then quickly ramping up to 50/50 (e.g., 50% of viewers will see version A, 50% will see version B).

19:00 Beware of launching experiments than “do not hurt,” there are feature maintenance costs.

20:00 Drive to a data-driven culture. “It makes a huge difference. People who have worked in a data-driven culture really, really love it… At Amazon… we built an optimization system that replaced all the debates that used to happen on Fridays about what gets on the home page with something that is automated.”

21:00 Microsoft will be releasing its controlled experiments on the web platform at some point in the future, but probably not in the next year.

21:00 Summary

Listen to your customers because our intuition at assessing new ideas is poor.
Don’t let the HiPPO drive decisions; they are likely to be wrong. Instead, let the customer data drive decisions.
Experiment often create a trustworthy system to accelerate innovation.

Justin Hunter
Founder and CEO
Hexawise
“More coverage. Fewer tests.”

Related: Statistics for Experimenters – articles on design of experiments – Combinatorial Testing for Software – Combinatorial Testing – The Quadrant of Massive Efficiency Gains

Curious Cat Management Improvement Institute is dedicated to furthering the education, professional growth and development of current and future managers and the practice of management in organizations.

That is quite a vast aim. We do think we can provide a useful service by connecting those interested in improving the practice of management with resources that will help them do so successfully.

We are building a web site to share resources (books, articles, webcasts, blogs…) on many management improvement topics (including: respect for people, systems thinking and data. I will be spending a fair amount of time the next few months adding to those pages.

Please let me know if you have comments, questions or suggestions for how to make the Curious Cat Management Improvement Institute useful to those trying to improve the practice of management. We believe there is a great deal of desire to improve how our organizations are managed. But decade after decade pass and the progress in how we manage organizations is very disappointing.

We believe we can help improve how we manage organizations. It is certainly a large task. But it is a worthwhile task and I believe an achievable one if we can think clearly and continually improve as we strive to improve the practice of management over the long term.

Some aspects of managing an organization well are quite challenging. But some aspects really are not and yet we still often fail to do even those aspects well. Bill’s suggestion to ask the customer what they want is one such area which offers an easy path to significant improvement. Though even such a simple concept is a bit challenging to adopt if the management culture doesn’t have an understanding of the organization as a system.

In the first part of this part of the seminar Bill talks about the value of talking to those in your organization that take what you produce (a product or service) and use that in their work as your organization continues to work to deliver to the “end users” (final customers). This seems so simple and unlikely to provide value but it is surprising how often what one part of the organization believes are the most important aspects of their output to others is not actually accurate.

Bill discusses how focusing on improving quality is the best way to reduce costs and increase productivity. Improving processes to reduce waste reduces costs and increases value to the customer. With costs decreasing prices can be lowered, providing more value (better products and services) to customers at a lower price. That leads to increasing sales which provides more work for more employees (see the Deming Chain Reaction).

Bill also talked about new jobs to do to work in a new way:

• Look for what could be improved. Then work on improving how the organization functions.
• Continual improvement of processes.

The focus is on continual never ending improvement. This is a much different mindset than focusing on what is most broken right now and how can we put in place some band-aid to allow things to keep working as they have but stop the visible issue as quickly as possible.

Both human (how to work effectively together, how to get people communicating, respect for people) and technical (problem solving tools, statistical methods, scientific approach) skills are needed to create a continual improvement culture.

Bill Hunter, in the seminar:

If you give these organizational development people just a little bit of help about statistical tools for tackling the problem, analyzing the problem, figuring out what it is and how to solve it their power can be multiplied many times over.

And if you give the statisticians and the technical people some help with organizational development skills the power they can have is multiplied enormously. It is incredible what can happen.
…
It’s joining those two in a synergistic way that can just produce amazing results. It is a skillful blending of these two.

Also

Around all these processes is potential information. And you can either use it or not use it. And in most places just waste it.
…
There is valuable information just running down the drain. If you have everybody turned on and looking at process and saying how can we make them work better, you need to get some information, you need to get some data. That is the way you actually solve problems.

See: Managing Our Way to Economic Success by William Hunter, which includes the quote:

Related: What Could we do Better? – Revealed Preference – Quality of the Entire Customer Experience

There is a bit of presentation that is outdated (mainly about general economic conditions and specific business conditions from the 1980s). If you don’t have the patience to sit through that (I find it useful but I can understand some people won’t want to listen to that part) just skip to about 21 minutes in and you will get great ideas of improving the management of our organizations today. You may even want to skip to about 36 minutes into the video if your attention span is less open to taking in a bit of background material.

In the section of the talk included here Bill discusses:

• Bigger Picture (economic conditions and history)
• Introducing Deming (as a person)
• Quality Improvement as Driver

US companies seem to be like a train. Up at the front there is the engine and that is where the motor is and this is what pulls the train along. The big bosses tell everyone what to do and then everybody does it…

In Japan, we like to run things differently. The engine is not just up in front and we all follow. We want to have all the employees have their own motors, start their own engines and that is much more powerful.

One would hope the model of designing business to treat employees with respect and enable everyone to think and act based on their knowledge and function would be widespread 40 years after this presentation. While I do think some businesses have learned to be less driven by a few bosses telling everyone how things must be done, overall USA businesses still fail to use the brains and drive all of their employees possess. See: Managing Our Way to Economic Success by William Hunter.

Business and Labor—from Adversaries to Allies by Donald Scobel was published in 1982 (I believe this is the article Bill referred to in the talk, though I could be wrong about that). In response to the article Russell Schrader, field representative AFL-CIO, wrote:

The present system in the US has not provided for any meaningful contribution from the workers themselves to improve the methods of production and the quality of work life. They have no opportunity to exercise their judgement, imagination, creativity or versatility in ways that could contribute to their productivity and sense of dignity. It is no surprise that they undergo frustration and discouragement, feelings certainly not apt to contribute to their efficiency. The solution to our productivity problem is the necessity for management and labor to recognize the intrinsic value of the human being.

In a response to that quote, Donald Scobel wrote:

In the last 4 years while researching the article, I have heard employees say over and over again in their own vernacular: “I want to contribute more than the organization will let me.”

Links to items mentioned in the presentation: text of the 1950 Dr. Deming talk in Japan – My First Trip to Japan by Peter Scholtes – “Building a Quality Movement,” with E. Chacko, August, 1972, Quality Progress (Bill mentioned talking to Deming about building a country-wide quality effort before bill spent a year and half as a professor in Singapore)

When the Japanese talk about quality control a better translation into English would be excellence. It is a much more all encompassing idea than what we think of as quality control.
…
When the Japanese talk about quality, it means not only the quality of the product, the quality of the processes producing the product, the quality of the designs that go into processes and the product, quality people, quality systems, quality everything. Quality service, just quality through and through everywhere. What they are really talking about is a new way to manage and run organizations.

Ishikawa talks about it as a thought revolution for managers. And that is what a lot of visitors to Japan have just missed.

Bill ends this portion of the presentation by saying “Quality is a journey, not a destination.”

Related : Interview of Bill Hunter, Brian Joiner and Peter Scholtes on Better Management Practices – Deming and innovation – What is Total Quality Control the Japanese Way by Kaoru Ishikawa – Bill Hunter and the Quality Movement (by George Box)

Brian Joiner, 1984 by Bill Hunter

We often saw things similarly. We could see things that we thought should improved and liked to focus on actually making improvements. Doing that is quite a bit more difficult than just pointing out all the problems that exist (in management and in society in general).

Brian was one of the people that best captured the desire and ability to make positive change, in my experience. Sometimes that means making compromises that will lead to actual improvements. I really enjoyed talking about ideas with him. He and Peter Scholtes were very similar to my father in their desire to improve people’s lives and the willingness to do the work to realize those improvements. It is very difficult to do.

His book, Fourth Generation Management, is one of the top handful of books I most recommend for those interested in improving the practice of management in their organization.

I have posted about Brian previously on this blog including:

• Interview of Bill Hunter, Brian Joiner and Peter Scholtes on Better Management Practices
• Dangers of Forgetting the Proxy Nature of Data
• Distorting the System, Distorting the Data or Improving the System (my post on The W. Edwards Deming Institute blog)
• Distort the System

My life is much richer for having known Brian. Many people’s lives are better due to the work Brian did during his lifetime. And more people’s lives will be better as his ideas, in Forth Generation Management and elsewhere, are applied in the future.

Related: Brian Joiner’s obituary (2023) – quotes by Brian Joiner – articles by Brian Joiner

We need to work together to optimize the system as a whole, not to seek to optimize separate pieces… Optimizing separate pieces destroys the effectiveness of the whole. For the organization to work well as a whole, the components must work together.

– Brian Joiner, Fourth Generation Management

blame the process not the person. We need to ask, “how did the process allow this to happen?”

– Brian Joiner, Fourth Generation Management

I was working to improve the processing time for court orders of child support (in the retirement system for the USA federal government). The time to process the court orders was taking far too long (longer than legally allowed). The first process improvements took place in the office (there were many easy ways to improve the process). In a few months things we finally starting to be under control and it was obvious the results were still far from acceptable. Looking at the whole process the time delay due to our office had been nearly entirely eliminated.

However the total time was still far too long. That time was not under our control, so how could it be our problem? Just because you do not control a portion of the process does not mean you cannot influence those results. By using a system that was already in place (but was used very rarely) to have the mail go directly to us rather than to a central mail processing location I was able to improve the process by more as any internal improvements.

Madison County courthouse in London, Ohio (Wikimedia image).

The delay from getting mail forwarded to our location from the central mailing location was many weeks long. I was able to have the orders for child support sent directly to us. Thus I was able to greatly improve the results with an improvement that was seen as not part of our system (the mail system delay before we received the orders). There are often ways to make improvements that are (or seem) outside of your area of responsibility and control.

By expanding the system view and looking at the results of the entire system it is often possible to find improvements that are not possible by only looking at “your” system. These changes can sometimes be more challenging to accomplish as they may require convincing others to make changes.

Related: Quality of the Entire Customer Experience – The Importance of Critical Thinking and Challenging Assumptions – Actionable Metrics – Good Project Management Practices

Yes, it isn’t hard to figure out Stephen Curry is a far better basketball player than some bench warmer. But trying to value some non-world-class-superstar is extremely difficult. Yet we have many people that think they can provide a great assessment of exactly what rating their people deserve. If someone is really able to judge people that well they should move into the front office of a sports team because they would pay a huge amount for such talent.

Mural at the Smith Samlanh Education Center in Phnon Phen, Cambodia

When you understand the challenges with evaluating a complex system it isn’t hard to know that evaluating individuals is not easy. Much of the evidence of individual “performance” is so dependent on impacts within the system that are totally out of even the individual’s influence. Yet it is easy to find numbers within a complex system that can be used to argue for or against an individual’s performance.

The contributions any individual brings to an organization is largely dependent on the system in place (see: 94% Belongs to the System).

Related: Expand Your Circle of Influence – Righter Performance Appraisal – Performance without Appraisal

Feeds of my previous management blog posts allow you to subscribe and receive daily updates on management improvement ideas. So even though I am not posting many new posts here anymore you have several options to get you more frequent management improvement content delivered to you. The feeds also allow you to choose to see items on additional topics (investing, engineering, travel, etc.).

John Hunter at Zion National Park, Utah, USA. One of the things I have been doing while not writing blog posts here is hiking in national parks (including Zion National Park earlier this year).

RSS feed options:

• Top Curious Cat posts (one post each day, from those I have selected as top posts)
• Curious Cat Management Improvement Blog (likely several posts a year, based on recent experience, as new posts are added to this blog)
• John Hunter management improvement posts (one post each day from my previously published posts on management)
• Curious Cat history – selected posts by John Hunter on topics including management, investing, travel, engineering… 2 new items are added to the feed every day from my previously published blog posts.

[updated to add link to my Mastodon profile]

Some people like to use Twitter as a pseudo RSS feed (getting tweets with links to interesting blog posts and articles). If that is what you would like to do I have two options: @aJohnHunter and @curiouscat_com. [I removed the links here to encourage using Mastodon which is where I am active now. I am leaving the details for Twitter; over time I expect Twitter will drop many of the poor policy changes they have made and return to offering a usable service. We will see if Twitter does actually address the many recent failings over the next few years. In any event RSS provides great value without the drawbacks on any social network service.]

See where else I can be found online: John Hunter online. Each of the blogs I author also allow you to subscribe to RSS feeds for their specific content.

I also have 2 YouTube channels that you may enjoy: Curious Cat Management and Curious Cat Travel.

In this interview Bill Hunter describes how results are made up of the impact and interactions of many variables. Many of those variables we don’t know about or account for. What we normally do is try to figure out the most important variables for processes and then experiment with those variables to find the best options given what we are trying to achieve.

Often the description of what is going on in such cases is that there is arbitrary error or random variation that influences the final results. What Bill discusses in this interview is that what is seen as arbitrary or random is often identifiably caused by specific variables. But often we don’t know what those variables are or how they are varying while we are getting different results over time.

He discusses how many research efforts seek to find the most important 2 variables and create a model based on those 2 variables to predict results. Even in PhD level research that is often done. He then discusses how to deal with other important variables.

He discusses the real world problems businesses must face in creating solutions that work.

If they are going to sell the product in Mississippi, and they are going to sell in Arizona and North Dakota, they have to have a robust product that will work in all these different conditions… It is not good enough for them to have a model that works sometimes… they’ve got to probe deeper and learn how relative humidity affects things and build that into the whole system in a different kind of way… they have to try and dig out the effects of these other x’s

So the business has to figure out the impact of many more variables in order to create reliable and robust products and services. This example is about variables that impact the use of the product by a customer, but the same concept applies to processes within your business.

For some processes it is enough to know a couple important variables and have an understanding of how they interact to impact results. Often though problems are created because the organization doesn’t learn enough about variables that can have a substantial impact on results and therefore feels blindsided by poor results. In some of those cases they were blindsided not by unforeseeable random factors but by variables they should have learned about. And then based on that knowledge designed their processes to take into account the potential impact of variations in that variable.

Bill also discusses the issues with how statisticians often approach analysis where the take the model (often based on only two important variables) and then use “experimental error” (“which we endow with all kinds of wonderful mathematical properties”) to explain the variation in results.

Result = f(x₁ + x₂) + g (x₃ + x₄ + x₅ + x₆ + …)

is seen as just:

Result = f(x₁ + x₂) + error

There is some justification for some of these assumptions [around normal error] because errors when you look at them do tend to be normal. However, they don’t tend to be independent, which is a key assumption.
…
If you really want to understand a system then you keep studying and studying and studying and moving some of the x’s in the g function to the f function. That is when you learn, and that is how you reduce variability, because this is where the variability comes from because some of [the items in g function] jump around.

The g function includes all those variables that are shown in one way of viewing a model as “random error.” So by learning about your system you discover move variables that are important and should be considered and managed and move them into variables that are considered in your model. By taking account of those variables that moved from function g to function f you can design the system to reduce variability. For example, if some measure of the input you receive from the supplier impacts the final result it may be possible to adjust a couple of variables that you have control over to adjust for that variation in order to get the result you wish to achieve.

In a service example, it could be if more than 1 of the senior level engineers are pulled off a project then it is necessary to adjust the team in some way (add a senior level project manager or add an extra person to the team)… in order to maintain the expected result of the team. The concept (learning about the processes and systems in your organization and knowing how variables will impact the results) is applicable to many situations. It is often easier to understand examples in manufacturing production but the concept is not limited to that domain.

Bill ends the interview with some thoughts on the challenges of learning and experiments in the social sciences.

In the video he is proposing this view (the g function with many x variables rather than just a catch all error function) would be a better way to introduce variation to statistics students. It seems to me it is also a better way for those interested in improving processes in their organization to understand variation.

Related links:

• More information on variation and interactions
• Life and Legacy of William G. Hunter web site
• Bill Hunter videos
• More Curious Cat videos

Part two of this interview: Improving Quality and Productivity in Organizations.

Related posts: Root Cause, Interactions, Robustness and Design of Experiments – The Art of Discovery by George Box – 200,000 People Die Every Year in Europe from Adverse Drug Effects, How Can We Improve? – Common Cause Variation