We have all been there. You dive into a new book or head to a conference/workshop/course and come out all fired up about a new evaluation method. But when you get back to the real world, applying it turns out to be harder than you thought!

What next?

Developing expertise and competence takes a lot more than just a one-shot training effort. And the journey to expertise differs for fairly straightforward technical skills (with quite specific guidance and steps to follow) compared to methods and approaches that need to be adapted to the situation, and for novice evaluators learning basic skills compared to experienced evaluators learning a new skill.

This year, Patricia and Jane are teaming up at the AEA conference to explore this issue with you.

In this flipped conference session, we invite participants and evaluators, evaluation managers and evaluation capacity developers around the world to build and share knowledge about what can be done to support people to apply new evaluation knowledge in practice.

Together we will explore these questions:

Q1. What are the different options for supporting people to learn how to actually apply new methods?

Q2. Which options suit which situations – and what needs to be considered when matching options to situations?

Q3. How can these options be implemented well?

Here’s how to participate:

If you would like to be part of this session, live or virtually, here’s what to do:

1. READ

Unlike a traditional conference session, we will be sharing materials beforehand for you to read, rather than using our face to face time to read them to you.

For each of the questions there is some brief material to read – with links to more detailed information to explore when it’s convenient

Q1. What are the different options for supporting people to learn how to actually  apply new methods?

Check out the list of options (on the BetterEvaluation site) for developing evaluation capacity.

Are we missing any? Suggest them by commenting on the page.

Q2. Which options suit which situations – and what needs to be considered when matching options to situations?

For an interesting list of factors to consider in the uptake of innovation (including characteristics of the innovation, the person adopting it, and their organization), check out this resource, by Trisha Greenhalgh and colleagues, which presents a conceptual framework from a systematic review of innovation in service organizations.

Are there additional factors to consider when adopting an evaluation innovation?  What are the implications of these factors? Add them in a comment here or on the BetterEvaluation evaluation capacity building page.

Q3. How can these options be implemented well?

Our page of examples will be progressively added to between now and the date of the conference session.

Do you have questions or comments about any of the examples? Add them as a comment on the page.

You can also read more about the different options, including exploring additional resources.

2. SHARE AN EXAMPLE

Do you have an example from your own experience of how you were supported to apply your new knowledge, or how you worked with others to do this for them?

Want to see an example of an example first? Jane has shared one already on the BetterEvaluation site, and there will be more to come.

>> Please share the details through this form. You can choose to identify yourself or to remain anonymous, and to attend the session or to contribute your example to be discussed by those at the session.

3. COME TO THE SESSION

• When? Sat, Nov 11, 2017 (11:15 AM – 12:00 PM)
• Where? Thurgood Marshall East, Washington Marriott Wardman Park.

We’d love to see you at the session.  If you have contributed an example, you will have an opportunity to respond to questions and comments that have been submitted and the questions from participants.

If you haven’t been able to do the preparation for the session, you are welcome to join the outer ring of the ‘fishbowl’ for the session.

And if you’re unable to attend in person, you can follow the action through twitter (#eval17 is the hashtag for the conference, and you can also follow Jane and/or Patricia directly).  To convert to your local time, use this converter.

4. PARTICIPATE AFTER THE SESSION

We will share the outcomes from the session through BetterEvaluation, including updating our list of options, and adding examples where permission has been given.  We will welcome your thoughts and comments after the session.

Please be in touch with us if you have any questions about the process.

I did love Michael Scriven’s analysis in his 2013 keynote to the Australasian Evaluation Society in Brisbane, where he remarked that the academic theorists are always considered the high priests of any discipline; the pure experimentalists closest/2nd tier; the “applied” folks, including the evaluators, have always been the almost-outcasts, their work sullied by getting their hands dirty in the real world.

But, as Scriven says, this attitude (and hierarchy) will be the death of the social science disciplines themselves.

Why? Because the most important questions any discipline ever asks and answers are the applied – especially the evaluative – questions. Like:

• What’s the most effective way to bring an economy back from a recession?
• What’s the most powerful way to transform social, educational, cultural, environmental, & economic systems so that they are sustainable and support thriving communities/economies/individuals?

The more these “applied” avenues of thought and inquiry are dismissed, the more a discipline relegates itself into the annals of irrelevance. Eventually resulting in “death by irrelevance” (of the discipline).

The only salvation:

• The applied researchers & evaluators should be at the “core” of any discipline.
• Theorists as satellites in service of the applied work.

But let’s take Scriven’s idea another step.

The theorist, the implementor, and the evaluator

Let’s start with the first two …

The theorist comes up with a notion about how the world works, or could be made better.

The “implementor” is seen as merely the person who puts the theorist’s brilliant ideas into practice.

What people hugely underestimate, though, is that just as much intellectual horsepower has been plunged into thinking through how exactly to contextualize an abstract notion and bring it to life in the real world with tools, systems, ways to persuade people and help them understand, support to help them do it, how to explain it to hostile stakeholders, and (of course – some of the evaluator’s piece) how to know in real time what’s working, what’s not, how we know, and what needs fixing first and fast!

Practical application (implementation) is in fact theory creation in its own right. From theory-based practice to practice-based theory. Adding to the theory what the theoretician never thought of because they don’t have that grounding in how it actually works on the ground. Heck, a lot of great theory is built from the ground up – how do you think we invented theory in the first place?

The implementor is the chef who takes the food theorist’s recipe, tries it in the kitchen, and works out how to bring out flavors to die for! The result is not just a vastly improved recipe; it now has the crucial added dimension of know-how (general and local), how to put “the twist” on the idea in just the right place to give it the edge.

The evaluative piece is just as important. It’s not just how we track things, people. It’s part of how we bring the theory to life in the real world. It’s how we show people “This is what success looks (and tastes) like when it happens; this is how you’ll know.” It’s about shared vision, something that guides the chefs, the sous-chefs, and the entire kitchen; it helps them visualize and therefore create it in real time, to the absolute delight of the clients.

Why do we privilege certain kinds of knowledge and insight creation over others that add just as much to the mix?

What happens when we do?

We get recipe books that look impressive but produce substandard cuisine. They knew too little about the nuances of the ovens or the local ingredients, or too little about the clientele and what they really yearned for.

Theorist: We love you and your brilliance, but you can’t make magic on your own.

Your implementor and your evaluator are brimming with just as much genius, just a different variety from yours.

And that’s what makes us lethal together.

Dr. Jane Davidson and educational systems change expert Joanne McEachen explain how sometimes the obvious solution is not in fact the right solution at all.

Even if it’s part of what’s needed, there are systemic issues in play that must be addressed as well.

Powerful insights to share with leaders you work with, whatever the sector.

Looking for an easy-to-grasp and much more compact alternative to rubrics? Try a minirubric!

A minirubric is a cross between a rating scale and a short rubric.

Hot tip #1: These aren’t an alternative to careful evaluative reasoning informed by the right mix of evidence, but (like full-size rubrics) they can be very useful to help get you there by focusing the discussion within the evaluation team and/or with stakeholders. Plus, they can make reports far easier to read and understand.

I developed a bunch of minirubrics recently for a participatory evaluation I was facilitating. The participating stakeholders were all interested in discussing the evidence and talking about what was good and what wasn’t so good. What I needed was to push them one more step so we could get to the evaluative interpretation of evidence. Not just talking about the strengths and weaknesses of things, but actually saying how good/bad/strong/weak the results were on balance, so that we could discuss that.

Hot tip #2: Create different minirubrics for different kinds of evaluation questions. For example, I developed a different minirubric for each of three high-level evaluation questions – one for evaluating the design and implementation of different program components; one for evaluating how good each of the outcomes was; and one for drawing conclusions about overall value.

Evaluating design and implementation:

Which parts of the [program] were the most informative, engaging and impactful*?

* “informative” = provided teens with useful insights they didn’t already have
“engaging” = presented in a way that got and held teens’ attention
“impactful” = positively influenced thinking, beliefs and/or intention to make safer choices

Hot tip #3: Make sure stakeholders are clear this isn’t just an ‘opinionfest’, and don’t fall into the trap of simply averaging the responses. The reason for asking them to make a rating is so that we can discuss the basis on which they came to that conclusion, including evidence and reasoning. Only after intensive evaluative deliberation together – guided by an evaluation specialist asking the tough questions and making sure the reasoning is sound – is an overall conclusion drawn.

Evaluating outcomes:

How well did the [program] provide teens with the knowledge and skills needed to make safer choices, and influence their attitudes, beliefs and intentions about safe and legal travel in cars?

Hot tip #4: Don’t forget, in order to call anything an ‘outcome’ you must show at least some evidence of a causal link. Need a cheap and simple way of doing that? Try building causation right into your survey or interview items.

Hot tip #5: People often think Michael Scriven’s definition of evaluation as “the determination of merit, worth, or significance” applies only to the overall program, policy, project, etc. Not true; you should be saying how good each one of your key outcomes is – as well as your program components, above). That’s what you need in order to step back and say how worthwhile the whole program (etc) was.

Hot tip #6: Use an even skinnier version of the minirubric to summarize your results across multiple findings in a readable way, e.g.

[Naturally, each of the ratings is backed by evidence later on in the report; this is just a short-hand way of summarizing some of the findings.]

Evaluating overall value:

How worthwhile was the [program] as an investment of time, effort and money to influence teens to make safer choices?

Hot tip #7: Your stakeholders generally only put effort into certain parts of the program, so the best thing to ask most of them is not the overall value of the whole program, but whether the results they’ve seen made it worth the effort they expended to achieve them, compared to what else they would be spending their time on. Use these stakeholder-generated ‘worth’ ratings alongside the rest of your evidence (e.g. Value for Investment analysis) to draw an overall conclusion about the value of the program. In your synthesis, give greater weight to whichever is more important in the grand scheme of things.

Want to learn more about evaluative rubrics?

Check out these resources:

• Evaluation Rubrics podcast: Jane Davidson interviewed on Adventures in Evaluation!
• Cool ideas from personnel evaluation: Evaluative rubrics
• Evaluation-Specific Methodology: The methodologies that are distinctive to evaluation
• Rubrics on the BetterEvaluation site
• Jane’s AEA eStudy webinar, 1st week of Dec 2014 (eStudy 051)
• Evaluation coaching to help you develop and use rubrics

Julian’s Value for Investment approach is a syncretic blend of economic and evaluative methodologies, including evaluative rubrics. As you can see from the picture below, it’s a topic of great interest in this part of the world, and there’s nothing we like better than puzzling over these things together!

A fun meeting of the minds in Auckland, New Zealand – seven kiwi evaluators puzzling over Value for Investment – [from left] Fiona Cram, Kate McKegg, Jane Davidson, Judy Oakden, Nan Wehipeihana, Julian King (and Kataraina Pipi on camera duty!)

What is Value for Investment, and how should we evaluate it?

Julian King, author of the Value for Investment ebook

Tena koutou katoa, I am a New Zealand evaluator and a PhD student at the University of Melbourne.

In the run-up to AEA Denver, I have posted this e-book on evaluating value for investment: http://www.julianking.co.nz/downloads/

Why Value for Investment? A term more commonly in use is value for money – but in my experience this gives undue emphasis to the money. Often, the most valuable resources and outcomes are intangible.

For example, more is invested in a social program than funding alone – intangible resources like cultural capital, knowledge and networks also have intrinsic value. While economics offers ways of representing intangible value in monetary terms, there are times when this is neither necessary nor desirable.

Is there a better way? Yes! Whatever methods we use, what we need is an overarching model to guide evaluation of VFI. My proposed model combines economic and evaluative thinking and has four components: VFI logic, context-appropriate valuing, evaluation logic, and evaluation with integrity.

I look forward to your feedback and questions, and hope to see you at AEA at my roundtable session: Getting to the real value in ‘value for money’.

Nga mihi,

Julian King