In evaluation we are often having to either translate evaluation jargon into plain language, or communicate intent and findings in languages other than our own.

It’s perhaps a little-known fact, though, how sometimes we even need English-to-English translation. Because of course what certain phrases mean in one culture may be quite different from what they mean in another.

Here’s a handy little interpretation guide for those trying to understand British English in particular. And, may we add, New Zealand and Australian English have many of the same nuances. Enjoy!

Source: http://www.scribd.com/doc/55551980/Anglo-EU-Translation-Guide

We’ve been both working on and researching about contracting in evaluation – serious issues about clarity in Terms of Reference, problems in terms of payment, and disagreements about whether ToRs should include an indicative budget.

So this joke seemed particularly apt as a metaphor for the formal and informal agreements that underpin any evaluation – and how misunderstandings can arise or be created.

An evaluator sits at a bar.

‘What’ll you have?’ asks the bartender.

‘A Scotch, please,’ replies the evaluator.

The bartender pours out the drink, passes it over, and says, ‘That’ll be five dollars.’

The evaluator replies, ‘What are you talking about? I don’t owe you anything for this.’

A lawyer, sitting nearby says to the bartender, ‘That’s right. In the original offer, which constitutes a binding contract upon acceptance, there was no stipulation of remuneration.’

The bartender is not impressed and tells the evaluator to finish the drink and get out.

The next day the evaluator returns. ‘What the hell are you doing back?’ says the barman. ‘I can’t believe you’ve got the nerve to show your face here!’

The evaluator replies, ‘What are you talking about? I’ve never been in this place in my life!’

‘I’m very sorry,’ says the barman. ‘But the resemblance is uncanny. You must have a double.’

So the evaluator says, ‘Thanks. Make it a Scotch.’

[somewhat adapted from http://www.manwalksintoajoke.com/bars]

Picture: http://www.flickr.com/photos/martialblog/4847174396/

I raised a few eyebrows last week when I mentioned the idea of Evaluation-Specific Methodology (ESM) as being an essential part of what defines us as a discipline.

Of course, a large proportion of people who identify as evaluators consider that evaluation is merely the application of social science research methods to support decision making – or something along those lines.

So, not surprisingly, someone asked (in the LinkedIn discussion that started me on this series of blog posts) what everyone else was probably wondering:

“What methodologies are those?”

The methodologies distinctive to evaluation are the ones that go directly after values.

Examples of evaluation-specific methodologies include:

• needs and values assessment
• merit determination methodologies (blending values with evidence about performance, e.g. with evaluative rubrics)
• importance weighting methodologies (both qualitative and quantitative)
• evaluative synthesis methodologies (combining evaluative ratings on multiple dimensions or components to come to overall conclusions)
• value-for-money analysis (not just standard cost-effective analysis or SROI, but also strategies for handling VfM analysis that involves a lot of intangibles, for example

I wouldn’t count the following as evaluation-specific: statistics or any of the standard research methods (interviews, observations, surveys, content analysis, or even causal inference methodologies). We clearly draw on these and use them a lot, but they are not distinctive to evaluation because they are not specifically about the “values” piece.

In other words, you could use these (non-evaluative qualitative and quantitative research methods) and still NOT be doing evaluation.

But if you are using ESM (evaluation-specific methodology), you sure ARE evaluating, i.e. drawing conclusions about quality, value, or importance.

And in fact, if you don’t use any ESM, you basically aren’t doing real, genuine evaluation. Either you are skipping the whole evaluative conclusions piece, or you are getting to it by logical leap (e.g. “I looked upon it and saw that it was good”). ESM is what allows us to get systematically and transparently from evidence about performance to evaluative conclusion, by weaving in the values (“how good is good”) piece.

It’s true that several disciplines use evaluation-specific methodologies (e.g. industrial & organizational psychology uses cost-effectiveness analysis). That doesn’t make them “not evaluation-specific” any more than statistics becomes psychology just because psychologists use it.

As I mentioned in the previous post, Michael Scriven and I have proposed a pre-conference workshop on Evaluation-Specific Methodology for AEA (in Washington DC) and a mini-workshop in the main program for AES (in Brisbane, Australia) this year, so if accepted we look forward to clarifying these concepts further!

My 2-day AEA workshop, Actionable Evaluation, has already been approved, and will cover (among other things) how to use evaluative rubrics to draw explicitly evaluative conclusions.

And, of course, see also the following books from Michael and myself!
[Already read these? Who would each one be particularly useful for? Please add your thoughtful evaluative reviews on Amazon - click the book to post one now!]

Related posts:

• Why “What’s the best tool to measure the effectiveness of X?” is totally the wrong question
• Credentialing – identifying the ‘core’ vs ‘specialized’ competencies
• The real values behind ‘value-undiscussable’ evaluation
• “No value-free”: The importance of visible values
• How good is a ‘good’ outcome?
• The importance of values for substantiating evaluative conclusions
• Why genuine evaluation must be value-based

Getting the definition of evaluation right is not simply a matter of having a popularity vote about it.

The fact that so many don’t see a clear difference between evaluation and other pursuits (such as research, monitoring, audit, organization development, management consulting) doesn’t mean that there isn’t one.

I just couldn’t resist commenting on this LinkedIn discussion, where the question was:

How do you define evaluation?

As a member of the CES National Council, a colleague and I have volunteered to try to come up with a definition of evaluation for the CES. So at this stage, we are using various lines of evidence to find out how people define evaluation or what makes research evaluation. Another suggestion I was given was to ask people to draw what evaluation means to them.

I am doing my part of the search through social media to see what kind of responses I get. I will do a content analysis, look for similarities and difference and maybe come up with an image depicting what I’ve heard. Another colleague is conducting a literature review with a grad student of hers, and the last is going to conduct a brief survey with users of evaluation.

What we noticed at our last CES national council meeting, was that we really should have a definition of evaluation on our website. But none of us could agree on a definition. Hence our exploration of different perspectives.

The fundamental difference is that evaluation asks and answers questions about the quality, value, and/or importance of things (design, implementation, outputs, outcomes, impacts, the project/program/policy/etc as a whole, and so on).

If we’re not doing that, we’re not actually doing evaluation.

And that has serious implications for our practice, and how well we can convey the value added of our entire profession.

I really enjoyed Michael Scriven’s webinar the other day (from the Claremont Evaluation Center), where he was talking about why it is absolutely essential we get clear on this:

“Roughly speaking, a profession becomes a discipline when it becomes more self-conscious and becomes clearer about the distinction between itself and its neighbors (related disciplines).

This means, for example, being clear about the methodologies that are *distinctive* to evaluation – in contrast with the methodologies that are also part of other related disciplines.

There are still people who identify as evaluators but who say you should *never* make a value judgement. They don’t like to be known as still hanging onto the value-free doctrine, but in fact that’s what they are.”

There are some more snippets of what he said on Michael Scriven’s Facebook page – and I’m hopeful Claremont Graduate University will post the recording of the webinar before too long.

Michael and I are proposing joint workshops on this topic (Evaluation-Specific Methodology) at both the American Evaluation Association and the Australasian Evaluation Society conferences this year. So, if accepted, we look forward to clarifying this for those who are interested.

But please, chime in with a comment to this post, or contribute to the discussion on LinkedIn.

What do you think?

Genuine evaluation is grounded in sound evaluative reasoning and evidence to justify conclusions drawn and insights surfaced.

One of the challenges we all face in doing sound reasoning is appropriately taking into account gender and cultural perspectives.

This week’s example may not be evaluative reasoning, but it’s a humorous example of how the reasoning and evidence used and the conclusions drawn can very much hinge on experience and perspective. And, how bringing in different user perspectives can help illuminate so much more about whatever we study or evaluate.

[Source: doing the rounds on Facebook; author unknown.]

A SPANISH Teacher was explaining to her class that in Spanish, unlike English, nouns are designated as either masculine or feminine.

‘House’ for instance, is feminine: ‘la casa.’

‘Pencil,’ however, is masculine: ‘el lapiz.’

A student asked, ‘What gender is ‘computer’?’

Instead of giving the answer, the teacher split the class into two groups, male and female, and asked them to decide for themselves whether computer’ should be a masculine or a feminine noun. Each group was asked to give four reasons for its recommendation.

The men’s group decided that ‘computer’ should definitely be of the feminine gender (‘la computadora’), because:

1. No one but their creator understands their internal logic;

2. The native language they use to communicate with other computers is incomprehensible to everyone else;

3. Even the smallest mistakes are stored in long term memory for possible later retrieval; and

4. As soon as you make a commitment to one, you find yourself spending half your paycheck on accessories for it.

(THIS GETS BETTER!)

The women’s group, however, concluded that computers should be Masculine (‘el computador’), because:

1. In order to do anything with them, you have to turn them on;

2. They have a lot of data but still can’t think for themselves;

3. They are supposed to help you solve problems, But half the time they ARE the problem; and

4. As soon as you commit to one, you realize that if you had waited a little longer, you could have gotten a better model.

The women won.