A free webinar on 16 September 10.30am – 11.30am CDT by Doug Hubbbard, “Measuring Risk – what doesn’t work and what does”, promises to address issues of appropriate measurement when dealing with intangibles and uncertainty.

Doug Hubbard, author of ‘How to Measure Anything: Finding the value of intangibles in business”, has developed the approach of “Applied Information Economics”, which promised to thoughtfully address uncertainty in developing estimates.

According to its Wikipedia entry:

AIE differs in several ways from other popular methods of decision analysis:

Unlike the accounting-style business case or cost benefit analysis, it does not rely entirely on point estimates of uncertain values. Since it uses the Monte Carlo method, uncertainty can be modeled explicitly.

Has anyone read the book? Or will attend the webinar? (As it is scheduled for 1.30am Friday Melbourne time, I don’t think I will be there).

Is it a useful new approach to measuring the hard-to-measure or an oversell on quantification? Is it only applicable to business or does it have applications in the government and not-for-profit sectors? Are the criticisms discussed in some reviews valid? Are there other texts on this issue that are better?

The webinar will focus on:

· The Problem – Why your method may be a ‘management placebo’ and why that is the biggest risk you have

· Problems that many methods ignore – and problems some methods introduce

· What Does Work – Studies reveal some methods show consistent, measurable improvements on the forecasts

· Examples of Real Improvements

· Overview of Applied Information Economics (AIE) Process

· Common Objections to quantitative methods and the misconceptions behind them

Webinar registration is at https://www1.gotomeeting.com/register/746702064

It is hard to imagine a more amazing journey to become an evaluator and international public speaker than the story of Tererai Trent. From humble beginnings in a cattle-herding family in rural Zimbabwe, where she was denied education because she was a girl and taught herself to read, Tererai has overcome incredible odds to achieve her dreams.

With her mother’s encouragement to aim high, Tererai wrote down her life goals on a scrap of paper and buried them in a tin box beneath a rock in a field where she herded cattle. Then she began systematically crossing those goals off as she achieved them.

From her poor village in Zimbabwe, Tererai left for America in the late 1990s and earned her bachelor’s and master’s degrees in 2000 and 2003 from Oklahoma State University. In 2009 she checked the last item off her list when she was awarded her doctorate – an Interdisciplinary Ph.D. in Evaluation from Western Michigan University.

Tererai’s work experiences, which include more than 8 years working for Heifer International as a Deputy Director of Monitoring & Evaluation, have taken her to five continents. Tererai worked with Michael Scriven, who supervised her doctoral studies at WMU, for five years in a major impact assessment that spanned 20 countries worldwide. Her main role was to ensure evaluative components such as gender, HIV, and cultural context were mainstreamed within each multicultural setting.

As a professional evaluator who now heads her own California-based evaluation consulting firm, Tinogona Consulting, she draws upon multiple perspectives of evaluation theory, practice and its utilization, as well as from her own extraordinary life experiences. Tererai brings together viewpoints from both the developing and developed world to provide a unique and very practical perspective on evaluation.

We are pretty sure Tererai is the world’s first evaluator to appear on the Oprah Winfrey Show. Make absolutely sure you check out her amazing story on the Oprah site in video form. It’s absolutely incredible. As New York Times columnist Nicholas Kristof said,

“Any time anyone tells you that a dream is impossible, any time you’re discouraged by impossible challenges, just mutter this mantra: Tererai Trent.”

Join us next week, starting on about September 12th, for a rich and interesting dialogue with Tererai!

Ban Dihydrogen Monoxide

(author unknown)

Dihydrogen monoxide is colorless, odorless, tasteless, and kills uncounted thousands of people every year. Most of these deaths are caused by accidental inhalation of DHMO, but the dangers of dihydrogen monoxide do not end there. Prolonged exposure to its solid form causes severe tissue damage. Symptoms of DHMO ingestion can include excessive sweating and urination, and possibly a bloated feeling, nausea, vomiting and body electrolyte imbalance. For those who have become dependent, DHMO withdrawal means certain death.

Dihydrogen monoxide:

• is also known as hydric acid, and is the major component of acidrain.
• contributes to the “greenhouse effect.”
• may cause severe burns.
• contributes to the erosion of our natural landscape.
• accelerates corrosion and rusting of many metals.
• may cause electrical failures and decreased effectiveness of automobile brakes.
• has been found in excised tumors of terminal cancer patients.

Contamination Is Reaching Epidemic Proportions!

Quantities of dihydrogen monoxide have been found in almost every stream, lake, and reservoir in America today. But the pollution is global, and the contaminant has even been found in Antarctic ice. In the Midwest alone DHMO has caused millions of dollars of property damage.

Despite the danger, dihydrogen monoxide is often used:

• as an industrial solvent and coolant.
• in nuclear power plants.
• in the production of Styrofoam.
• as a fire retardant.
• in many forms of cruel animal research.
• in the distribution of pesticides. Even after washing, produce remains contaminated by this chemical.
• as an additive in certain “junk-foods” and other food products.

Companies dump waste DHMO into rivers and the ocean, and nothing can be done to stop them because this practice is still legal. The impact on wildlife is extreme, and we cannot afford to ignore it any longer!

The Horror Must Be Stopped!

The American government has refused to ban the production, distribution, or use of this damaging chemical due to its “importance to the economic health of this nation.” In fact, the navy and other military organizations are conducting experiments with DHMO, and designing multi-billion dollar devices to control and utilize it during warfare situations. Hundreds of military research facilities receive tons of it through a highly sophisticated underground distribution network. Many store large quantities for later use.

It’s Not Too Late!

Act NOW to prevent further contamination. Find out more about this dangerous chemical. What you don’t know CAN hurt you and others throughout the world. Send email to no_dhmo@circus.com, or a SASE to:

Coalition to Ban DHMO
211 Pearl St.
Santa Cruz CA,
95060

Note: In case you haven’t figured it out “Dihydrogen monoxide” is H20 – water.

Are YOU a problem thinker?

(original author unknown – and we’ve made a few adaptations for evaluators)

It started out innocently enough. I began to think at conferences and parties now and then to loosen up. Inevitably though, one thought led to another, and soon I was more than just a social thinker.

I began to think alone – “to relax,” I told myself – but I knew it wasn’t true. Thinking became more and more important to me, and finally I was thinking all the time.

I began to think on the job. I knew that thinking and employment don’t mix, but I couldn’t stop myself.

I began to avoid friends at lunchtime so I could read heavy-duty evaluation think pieces from Michael Scriven, Yvonna Lincoln, and Rodney Hopson. I would return to the office dizzied and confused, asking, “Has anyone ever really questioned the power structures in this organization?” and “What about the cost analysis, people?!”

Things weren’t going so great at home either. One evening I had turned off the TV and asked my wife about what values might be applied by St. Peter at the pearly gates when he evaluated applicants for entry into different afterlife accommodations. She spent that night at her mother’s.

I soon had a reputation as a heavy thinker. One day the boss called me in. He said, “Skippy, I like you, and it hurts me to say this, but your thinking has become a real problem. If you don’t stop thinking on the job, you’ll have to find another job.” This gave me a lot to think about.

I came home early after my conversation with the boss. “Honey,” I confessed, “I’ve been thinking…”

“I know you’ve been thinking,” she said, “and I want a divorce!”

“But Honey, surely it’s not that serious.”

“It is serious,” she said, lower lip aquiver. “You think as much as college professors, and college professors don’t make any money, so if you keep on thinking we won’t have any money!”

“That’s a faulty syllogism,” I said impatiently, and she began to cry. I’d had enough. “I’m going to the library,” I snarled as I stomped out the door.

I headed for the library, in the mood for some Saville Kushner, with an Eleanor Chelimsky keynote blaring on my ipod. I roared into the parking lot and ran up to the big glass doors… they didn’t open. The library was closed.

To this day, I believe that a Higher Power was looking out for me that night.

As I sank to the ground clawing at the unfeeling glass, whimpering for Bob Williams’ latest stuff on systems concepts or Doug Reeler’s three-fold theory of social change, a poster caught my eye. “Friend, is heavy thinking ruining your life?” it asked. You probably recognize that line. It comes from the standard Thinker’s Anonymous poster.

Which is why I am what I am today: a recovering thinker. I never miss a TA meeting. At each meeting we watch a non-educational video; last week it was “Porky’s.” Then we share experiences about how we avoided thinking since the last meeting.

I still have my job, and things are a lot better at home. Life just seemed… easier, somehow, as soon as I stopped thinking!

Patricia just finished running two courses for The Evaluators Institute in Washington DC on using program theory and logic models for evaluation, and analyzing qualitative data analysis, and a series of workshops on purposeful program theory for the Australasian Evaluation Society, and workshops with ARTD Consultants (Sydney) and the NSW Dept of Ageing, Disability and Home Care.

Upcoming events:

Tuesday 31 August Jane will lead a workshop on “Actionable answers for real-world decision makers: Real evaluation nuts and bolts that deliver” at the conference of the Australasian Evaluation Society in Wellington, New Zealand (workshop is full).

Tuesday 31 August Patricia will present on ““Between theory and practice: formulating a research project on evaluation” and contributing to a round table on “Epistemological foundations of evaluation methodologies” at the International Summer School on Public Policy Evaluation Research, in Lille, France.

Friday September 3 Jane is also giving a keynote for the Australasian Evaluation Society conference (in Wellington, New Zealand) entitled Extreme Genuine Evaluation Makeovers (XGEMs).

Thursday 7- Friday 8 October Patricia, together with Dr George Argyrous and Associate Professor Jo Baulderstone, will present a two-day workshop on “Evaluation for Public Sector Managers” for the Australian and New Zealand School of Government in Sydney, Australia.

Wednesday 10 November Jane will lead a full-day workshop on “Getting Actionable Answers for Real-World Decision Makers: Evaluation Nuts and Bolts that Deliver” at the conference of the American Evaluation Association, in San Antonio, USA.

Sunday 14 November Patricia will lead a half-day workshop on “Purposeful Program Theory” at the conference of the American Evaluation Association, in San Antonio, USA.

We both look forward to engaging face to face with the people who have followed and/or joined the discussions here on the Genuine Evaluation blog!

Australian researchers will conduct a world-first study to find the optimal treatment for the debilitating, and often deadly, disorder anorexia nervosa. Around one in five people, usually women, who develop the eating disorder would die as a result, says Associate Professor Susan Byrne from the University of WA’s School of Psychology.

She said many treatments had emerged over the years, from different models of counselling and therapy along with acute hospital care, though there was little evidence to show which approach was best.

“Currently, there are a whole range of treatments and we don’t have any evidence that one is any better than the other,” Dr Byrne said on Friday.“… We’re really trying to gather some evidence so we can say to people, with confidence, that we have evidence that this or that treatment is helpful.”

Dr Byrne is seeking around 200 adult volunteers who have an eating disorder and who are based in Sydney, Adelaide or Perth. Participants will receive ten months of free treatment – in one of three different front-line therapies developed independently and used by some to tackle anorexia in different parts of the world.

The research will compare Enhanced Cognitive Behaviour Therapy, developed at Oxford University; Maudsley Anorexia Nervosa Treatment for Adults (MANTRA), developed at London’s Maudsley Hospital; and New Zealand’s Specialist Support Clinical Management for Anorexia Nervosa.

I would have hope that after all this time and effort developing evidence-based policy and practice  it would be widely understood that:

1. One size rarely fits all. Universal “one size fits all” prescriptions of practice are very often inappropriate for some sub-groups.
2. One study cannot answer this question. A a single study, no matter how large or well-designed and implemented, will have serious limitations compared to a synthesis of multiple studies that tries to take account of all credible evidence about comparative effectiveness and situational appropriateness.

Perhaps it is due to the way the study has been reported, rather than the way it has been actually proposed.  Let’s hope that is the case.

A few snippets to whet your appetite:

WASHINGTON (AP) The results of a new survey conducted by pollsters suggest that, contrary to common scientific wisdom, correlation does in fact imply causation. The highly reputable source, Gallup Polls, Inc., surveyed 1009 Americans during the month of October and asked them, “Do you believe correlation implies causation?” An overwhelming 64% of American’s answered “YES”, while only 38% replied “NO”. Another 8% were undecided. This result threatens to shake the foundations of both the scientific and mainstream community.

…

Others are excited because this new, important result actually gives insight into why the result occurred in the first place. “If you look at the numbers over the past two decades, you can see that Americans have been placing less and less faith in the old maxim ‘Correlation is not Causation’ as time progresses.” explained pollster and pop media icon Sarah Purcell. “Now, with the results of the latest poll, we are able to determine that people’s lack of belief in correlation not being causal has caused correlation to now become causal. It is a real advance in the field of meta-epistemology.”

Of course, the article also includes views from the scientific community to add credibility and depth:

The scientific community is deeply divided on this matter. “It sure makes my job a lot easier.” confided neuroscientist Thad Polk. “Those who criticize my work always point out that, although highly correlated, cerebral blood flow is not ‘thought’. Now that we know correlation IS causal, I can solve that pesky mind-body problem and conclude that thinking is merely the dynamic movement of blood within cerebral tissue. This is going to make getting tenure a piece of cake!”

Anti-correlationist Travis Seymour is more cynical. “What about all the previous correlational results? Do they get grandfathered in? Like, the old stock market/hemline Pearson’s rho is about 0.85. Does this mean dress lengths actually dictated the stock market, even though they did it at a time when correlation did not imply causation? And what about negative and marginally significant correlations? These questions must be answered before the scientific community will accept the results of the poll wholeheartedly. More research is definitely needed.”

The full article may be viewed at devpsy.org, who note that this column was produced originally in the Happy Hour Archives, a collection of cognitive psychology humor.

It’s become my benchmark for visible, sincere and enthusiastic support from senior management for evaluation.

This week that manager, Dr Bruce Kefford, now Deputy Secretary, Agriculture Research and Development in the Victorian Department of Primary Industries, spoke to the meeting of the Victorian Branch of the Australasian Evaluation Society about developments in evaluation since that time. He is still enthusiastic about evaluation, and talked about the strategies they have put in place to prevent or reduce the common problems of evaluation such as goal displacement and data corruption.

He also referred to the 7 lessons we drew from the first 5 years of their work to build organizational evaluation capacity and capability (capacity is like having a water tank; capability is having pipe connected to it so you can make use of the capacity – thanks to Bob Williams for the metaphor). These were published in the article Bron McDonald, Patricia Rogers, and Bruce Kefford Teaching People to Fish? Building the Evaluation Capability of Public Sector Organizations Evaluation January 2003 9: 9-29:

1. start small and grow evaluation;
2. address both supply and demand;
3. work top-down and bottom-up simultaneously;
4. use a theory of change behaviour;
5. develop a common evaluation framework, including a generic programme theory;
6. build knowledge of what works within the agency’s context; and
7. systematically and visibly evaluate each stage.

Looking back at the list, and thinking about many of the other organizations I’ve worked with over the years, I think many of these still stand up as good advice. I’m not so sure a generic programme theory is always helpful (they found they got more traction using a common template, but there are obvious risks if the template doesn’t suit all the types of programs).

What is missing is the very important strategy of trying to reduce the fear of evaluation by making it clear that identifying problems did not always leads to closing programs (maybe they could be improved), or where it did, that this did not always lead to losing staff (maybe they could move to another program).

Another issue that is not addressed is the importance of sustained effort over time. Some organizations were shocked to see that the process had taken 5 years (and of course is still continuing), and wanted instead a way they could just get it all done in a couple of months.

What do you think of the others? Are they visible in other successful efforts to grow evaluation capacity and capability?

Their television advertisements are based on 2 arguments:

1. there is no evidence this will work, in terms of reducing smoking among either current smokers or new smokers
2. it will add to the cost of running their businesses

Their statement of “what we stand for” states:

The government proposal to mandate plain packaging for cigarettes is the last straw.
Let’s be clear – we believe that reducing smoking is good for our community. But good policies require more than good intentions.
There is no reliable evidence anywhere in the world that plain packaging will stop people from taking up smoking, or help people to quit. But we do know that it will make it harder for us to run our businesses.

“No real evidence”

One of the ads argues strongly that policy must be based on evidence not just good intentions. Despite the appeal of this line, there is something not so convincing about the argument.

Well, there is evidence from experimental studies, such as a 2008 study published in the British Medical Journal.

If they are asking for evidence in terms of a longitudinal study of the actual effects of this when introduced as policy, it’s pretty hard to get this, as it has not been introduced anywhere.

And of course there is a self-perpetuating pattern to this – if this argument is run everywhere, it never will, making it impossible to produce the evidence.

“It will make it harder to run my business”

Another ad argues that the new policy would be damaging to business, without specifying how.

The spokesperson for the AAR, Sheryle Moon, interviewed on the ABC current affairs show Lateline, tried to explain how this could be the case:

SHERYLE MOON, ALLIANCE OF AUSTRALIAN RETAILERS: If I’m a small business owner, perhaps I’m a single owner operator, so I’m in my store, I’ve got plain packaging of cigarettes, I’ve got to put them away on a shelf – it’s difficult to do that. I can’t identify necessarily which product is which product.

PETER LLOYD: Why not?

SHERYLE MOON: Because it’s harder to see them, they’re not identified, they all look very similar. If I’m serving customers, it’s difficult for me to find the right product for the customer. All those increase my transaction times and make it more difficult for me to run my business efficiently.

PETER LLOYD: But the cigarette brand name would still be printed on the bottom of the package so you could still see, for example, Marlboro Light, on the packet, so why is that so difficult?

SHERYLE MOON: So I think the issue here, Peter, is more about that plain packaging is not a proven policy.

A more convincing argument comes from the webpage of the AAR, which reports the results of a survey (no details of sample size or type…):

What we do know about plain packaging is that it will make it harder for us to run our businesses.

A May 2010 Galaxy poll found:

	80 per cent of retailers believe the plain packaging policy would hurt their business
	81 per cent of businesses surveyed consider tobacco sales important to their business
	78 per cent of retailers believe their business will suffer and they may have to lay off staff if their customers turn to the black market for tobacco
	87 per cent of retailers believe small business in Australia is faced with too much red tape and regulation.

So… tobacco sales are important and they predict a plain packaging policy would hurt their business. I guess that would be by reducing sales – that is, by working?

The AAR is a new organization comprising The Service Station Association, Australian Newsagents’ Federation and National Independent Retailers Association, which is supported by British American Tobacco Australia Limited, Philip Morris Limited and Imperial Tobacco Australia Limited.

Some important contrasts presented but also one that doesn’t entirely align – tell the whole story but cut to the chase? Include activities, outputs and outcomes AND identify the issue of essence/determine what’s important? Um? Are both possible at the same time?

[see Irene's full response for more detail]

Managing these tensions and apparent paradoxes is, in my view, one of the things that really clearly distinguishes genuine from non-genuine evaluation.

When I used to teach evaluation to doctoral students in WMU’s Interdisciplinary Ph.D. in Evaluation, here’s what I insisted they do in evaluation reports, short homework assignments, essays, dissertations, and presentations (which, to give credit to the original source, was more or less what Michael Scriven used to insist we do when I studied with him at CGU!):

1. Even when there are numerous details and nuances that need to be reported, it’s still critically important to cut to the chase. This means present the most important points first – it shows you know which they are! The secondary issues come next. [This is the exact opposite of traditional academic writing, where we gradually lead the reader through the maze of details to the overall conclusion.]
2. On the issue of how much detail is enough/too much, the advice to tell the whole story (but don’t get lost in the details) means: tell enough of the story so that the overall messages aren’t misleading.

I’m also reminded of Eleanor Chelimsky’s advice this week about how to present findings in a way that won’t make the audience glaze over with the level of detail – this is from a post in this week’s riveting Thought Leader’s Forum (from AEA – it’s members only access, but membership is incredibly reasonable and access to these discussions (and much more) make it fantastic value for money!):

… But you had to observe Members of Congress closely.  In theory, for example, the legislators loved what they called “hard data,” but I found that their eyes frequently glazed over during my presentations of them.  I often ended up forced to present a really strong study in the form of anecdotes or case studies, which were, however, not an N of 1, but were representative of the study data and overall findings, and had the advantage of being entirely comprehensible to a congressional audience as well as the media.  I should also mention that there was a secondary penalty for NOT doing that:  not only did our message not get across, but we were seen as technocrats, not a good image for that audience.

For me, it’s taken years of experimentation and practice with different ways to cut to the chase and present key messages that various audiences can comprehend and remember, while at the same time telling enough of the nuanced story so as to make the main messages not overly simplistic or misleading. I’m by no means successful all the time, and I’m always on the lookout for insights and ideas. So, please contribute yours in a comment!