Photo: Phillippe Halsman

 

“Not everything that counts can be counted, and not everything that can be counted counts.” — Albert Einstein

It’s a familiar discussion in the evaluation world – the importance of getting approximate answers to important questions about what really matters, rather than being seduced into the “indicators” game where we measure the living daylights out of whatever is easiest to measure and then worship that.

Because, after all, quantification gives things a much higher truth value – right?

Right …

The business world in particular still struggles to get buy-in to a mixed method approach to evaluation.

There is a heavy emphasis on quantitative indicators such as financial/bottom line and the easy-to-measure such as web hits, etc.

A recent article in Forbes magazine by marketing specialist Susan Gunelius tries to get her marketing colleagues to think outside the box about the evaluation of marketing and include important, hard-to-measure outcomes such as the nature of engagement with the brand.

Susan suggests we should consider not just traditional Return on [Financial] Investment, but also:

Return on Impression – Eyeballs: the number of people who actually see your ad, marketing material, or other tangible marketing piece

Return on Impression – Perceptions: feelings for those brands and emotional connections to them

Return on Opportunity: the indirect marketing opportunities that it could lead to as people discuss it and share it across the social web (and offline) versus the time and monetary commitment that effort requires

Return on Engagement: the value of a positive online buzz about your brand, products, services, and business; how well your brand is performing in terms of building and sustaining relationships with both consumers and influencers

Return on Objectives: how well marketing efforts help a business move in the right direction to meet its long-term objectives (such as brand building)

With an evaluator’s lens on, perhaps we can suggest some improvements to this line-up? Please chime in with comments, and I will plan on a post on this topic as a follow-up.

The comments section of Susan Gunelius’ article shows just what an uphill battle it is in the business sector to get buy-in to any non-financial considerations. For example:

Sorry but in my opinion this smacks of a desperate attempt to cover up marketing’s tenuous or fictional link to real business metrics. Either marketing can contribute directly or indirectly to growth in a tangible way or it cannot. No amount of window dressing (fancy new handles, job titles, buzzwords) will help–in fact they only contribute to the perception that marketing is clinging to relevancy. Effective CEOs, CFOs and CMOs who “get it” have gotten past these arguments; they have made their determination on marketing’s relevance and effectiveness based on their industry and channels and have right-sized their marketing investments accordingly! — Bob London

Sound familiar?

As evaluators we often have to respond to those who believe that a mixed methods approach means we are ignoring the bottom line. [And actually, a lot of evaluations DO ignore the bottom line - but that's another story, and not an inherent flaw of genuine mixed methods evaluation.]

I did like Susan’s response to this and another “I agree” comment:

I think that’s the point though. Both you and Bob left comments on this post and Bob also tweeted it. Is the only value you’ll both derive from those comments the number of page views that this post gets (assuming every person who viewed this article also read the comments)? Or are there soft metrics that matter as well — in terms of audience perception and sentiment related to what you said in your comments and how those comments are discussed outside of this post? If you only track page views and retweets (which I’m assuming you don’t), you’re missing part of the story.

From an evaluation perspective, a few other points occur.

Financial outcomes are ‘lag’ indicators of success. They usually tell you quite a while after the fact whether there has been any change in the bottom line.

In general it’s better to also get an early read on what’s working or likely to work – the ‘lead’ indicators. These include, for example, how consumers react to or engage with the brand.

A lot of this earlier evaluation can actually happen prior to investing a huge spend in a particular marketing strategy.

This is one of the key tenets of the Balanced Scorecard approach, which is widely used in business.

The so-called ‘soft’ indicators of success are part of a causal chain that includes financial outcomes. As one of the commenters on the article put it:

Often – impressions > inquiries > leads > opps > $ … engagement can be around the loyalty constellation or opportunity around market share … we mostly use proxies anyway in linking soft to hard. — mperla

See? There’s often an important theory of change buried in all sorts of evaluative thinking!

But a very important point here is that following this causal chain is a key way to be able to attribute financial outcomes to the marketing activities (or, to estimate their contribution). Causal inference is notoriously difficult if you only track the downstream (long term) indicators.

Marketing is not just about advertising. A serious strategic approach to marketing involves truly understanding the needs and wants of consumers and building these right into the product or service development in the first place.

And if this groundwork has happened, then a logical place to go when looking at the effectiveness of advertising is to see how well it keys on solving that problem for them, or creating that value, or forging that emotional relationship with the brand.

An important message here is that what you sell (and therefore what we must evaluate) is not the “thing” (the product or the service) but the value it generates for consumers or recipients (and then, down the causal chain, to value for shareholders, or for society).

And that value may be solving a problem, making life better, or creating an emotional connection that produces enjoyment – or all three.

“I’ve learned that people will forget what you said, people will forget what you did, but people will never forget how you made them feel.” Poet, Dancer, Producer, Playwright, Director, Author Maya Angelou (born 1928)

Thoughts?

Implications for evaluation in your sector?

Or, for how we market our services?

There is as much for us to learn from marketing as we have for them to learn from us.

Business Cartoon by Andertoons

It’s a puzzling fact that one of the most costly and stressful organizational change strategies – restructuring – is virtually NEVER subjected to any serious evaluation.

The reshuffling of lines and boxes; the layoffs; the unwanted ‘domino effect’ turnover of valuable employees; the ripple effects on families; whether the intended improvements in focus, efficiency, productivity, and competitive advantage emerge; or whether they outweigh the costs and negative side effects … all subjected to little more than some armchair analysis by management.

Public and private, for-profit, non-profit, and government organizations spend hundreds of thousands of dollars (sometimes more) on working out the best restructure and implementing the change.

But for some unfathomable reason, no-one ever says “I wonder if we should see whether all that expenditure and upheaval was worth it” or “We should try and learn from this huge initiative so we can get it right as it unfolds and do it better next time.”

Or do they?

In 2005, four intrepid health researchers from Norway and Canada published a review of all the articles that evaluated reorganizations.

The following is their abstract, but it is well worth clicking through to the full article!

A surrealistic mega-analysis of redisorganization theories

SUMMARY

Background We are sick and tired of being
redisorganized.

Objective To systematically review the empirical
evidence for organizational theories and repeated
reorganizations.

Methods We did not find anything worth reading,
other than Dilbert, so we fantasized. Unfortunately, our
fantasies may well resemble many people’s realities.
We are sorry about this, but it is not our fault.

Results We discovered many reasons for repeated
reorganizations, the most common being ‘no good
reason’. We estimated that trillions of dollars are being
spent on strategic and organizational planning
activities each year, thus providing lots of good reasons
for hundreds of thousands of people, including us,
to get into the business. New leaders who are
intoxicated with the prospect of change further fuel
perpetual cycles of redisorganization. We identified
eight indicators of successful redisorganizations,
including large consultancy fees paid to friends and
relatives.

Conclusions We propose the establishment of ethics
committees to review all future redisorganization
proposals in order to put a stop to uncontrolled,
unplanned experimentation inflicted on providers and
users of the health services.

Old mistake in today’s article on European responses to austerity measures – here, as reported by Karen Kissane in The Age in Melbourne:

Meanwhile, in Greece, a country spiralling into poverty with more than half of its young people unemployed,

This seems to be a common misinterpretation. A Reuters report led with the headline:

Over half of Greek youth unemployed

A BBC report stated:

More than half of young people in Greece are unemployed …

Well. no, actually.  Unemployment rates are the percentage of those available for and looking for work who are unemployed.  Since a large proportion of young people are engaged in full time study, the proportion of the population who are unemployed is much less.

According to the European Commission’s Eurostat page “Statistics explained” on unemployment statistiscs, Greece’s youth unemployment rate has gone up from 25.7% in 2009 to 44.4 in 2011 and 49.3 in the last quarter of 2011 – but the ratio – that is, youth unemployed as a percentage of the youth population has only increased from 8% in 2009 to 13% in 2011.  Still a tragedy for those affected, but very different to the picture painted.

Discussions about public policy need to be based in accurate representation of the situation. This is a basic mistake that should not be made.

As evaluators, don’t we all just love the client who insists on the Rolls Royce evaluation with all its bells and whistles … but is prepared to pay for little more than a rickety bicycle?

Even better, don’t you love it when they spring this on you only after you’ve done a lot of extra work you understood you’d be paid for?

Well, here’s a little something to send in with your next proposal – after you make the shortlist and are in that all-important price negotiation stage. [We found this on the humor page of the Institute of Management Consultants.]

If you can’t see the video below (e.g. because you are reading this on the email feed), you will need to click through to the post on the Genuine Evaluation blog.

How important is content expertise for an evaluator?

We have discussed in a previous post the circumstances under which evaluators require not only evaluation expertise but a deep knowledge of the subject matter itself.

And in another (a Friday Funny) we pointed out how shortfalls in evaluation and cultural expertise are often even more fatal for good evaluation than shortfalls in content expertise.

But, let’s face it, there are some situations where dropping by with our methodologies, technology, and analyses just doesn’t go well without some fundamental content knowledge – not to mention at least a minimal focus on utilization (like, what does the client actually need to know?).

This gem is an unauthorized adaptation of one of those consulting humor classics (original author unknown), drawn from WorkJoke.com

Evaluation and content expertise

A farmer was herding his flock of sheep in a remote pasture when suddenly a brand-new BMW advanced out of the dust cloud towards him.

The driver, a young man in a Broni suit, Gucci shoes, Ray Ban sunglasses and YSL tie, leaned out the window and asked the farmer … “If I tell you exactly how many sheep you have in your flock, will you give me one?”

The farmer looked at the man, obviously a yuppie, then looked at his peacefully grazing flock and calmly answered “sure”.

The yuppie parked his car, whipped out his iPad, then he surfed to a NASA page on the internet where he called up a GPS satellite navigation system, scanned the area, and then opened up a database and an Excel spreadsheet with complex formulas. He sent an email and, after a few seconds, received a response.

Finally, he prints out a 130-page report on his miniaturized printer then turns to the farmer and says, “You have exactly 1586 sheep.”

“That is correct; take one of the sheep.” said the farmer. He watches the young man select one of the animals and bundle it into his car.

Then the farmer says: “If I can tell you exactly what your business is, will you give me back my animal?”

“OK, why not.” answered the young man.

“Clearly, you are an evaluator.” said the farmer.

“That’s correct.” says the yuppie, “but how did you guess that?”

“No guessing required.” answers the farmer. “You turned up here although nobody called you. You want to get paid for an answer I already knew, to a question I never asked, and you don’t know crap about my business…… Now give me back my dog.”