Results-Based Management Websites

Online RBM Training -2: The UN's Programme Performance Assessment in Results-Based Management

2012-01-31T22:17:00.006-05:00

--Greg Armstrong --

The UN’s Programme Performance Assessment in Results-Based Management, is a jargon-laden, time-consuming and partially out of date production that, whatever its original merits, is too frustrating to be useful today. There are more productive ways to spend your time.

Level of Difficulty: Moderate to Complex
Primarily Useful As: Difficult to say - a history of UN RBM jargon?
Most useful: Section 4-5 on data collection and reporting
Limitations: Out-of-date RBM language,jargon-laden, boring format

Background:

Online RBM training is unlikely to provide the hands-on practice and the opportunities for learning of group processes which are the foundation for effective Results-Based Management training and implementation, but some courses are definitely better than others.

After my December 15, 2011 post describing the University of Wisconsin’s effective interactive online results-based management course, I was asked by one reader for my opinion of other online RBM courses. The University of Wisconsin’s Logic Model course, is, in my opinion, excellent, making the most of interactive opportunities available in the absence of live training.

On the other end of the spectrum, however, is an online course available from the UN Office of Internal Oversight Services–Programme Performance Programme Performance Assessment in Results-based Management.

Google search results - Online Results-Based Management Training
CLICK image to enlarge

If we do a plain word search for "online results based management course" (without the quotes) on Google's .com search we will get, depending on the day, more than 80 million hits, and on page one of this search – at least as of January 31, 2012, the first and third results shown are for this specific UN RBM course.

Those search results may vary slightly given country-specific versions of the search engines, but Microsoft's Bing and Yahoo search return several million results on this topic too, again with this UN RBM course ranking very high in the search results.

The highest ranked result --with a 2011 date-- is UNESCO’s Open Training Portal, and if we click on the UNESCO link, it takes us to what appears to be the home of the course, the UN’s Office of Internal Oversight Services, which is number 3 on the search list. The course also appears as a link in some other UN documents.

So - how does this highly-ranked course stand up to scrutiny?

A disappointing use of the web

For the general development community this online RBM course is not worth the time it will take to complete it. With over 150 screens, it could easily take 3-4 hours to complete this, and most of us could make much better use of written manuals to get the same, and more detailed information, in a shorter period.

For UN staff, it is possibly of greater use than for the general public, other implementing agencies or national partners, but there are, even here, some limitations. It may appear, perhaps, unfair to criticise the lack of utility of this course for the general development community, when it was intended primarily as an in-house United Nations training tool when it was originally designed. The problem is, that for any user, UN or not, there is little that is compelling in the way this is put together.

Not an interactive course

This RBM course makes no use of the engaging possibilities of interaction that the University of Wisconsin used to advantage in an online Logic Model course designed at least 3 years before this UN course was published.

At the very lowest level of potential interactivity, the menu on the UN course is extremely limited. Readers can skip to one of 4 “sessions” but if we want to move to subthemes inside them, after quitting and then rejoining the course, for example, laborious clicking is required to move slide by slide forwards or back to potentially interesting topics.

There is some interesting material, here, but nothing you can’t find more easily in printed handbooks or electronic files. The content of this course is basically just material taken from other available online RBM Manuals and documents.

The exercises, where they occur, simply ask the viewer to download Microsoft Word documents and complete them.

The few tests of skill, such as matching technical terms such as “Impact”, “Objective” or “Indicator” with definitions either require the viewer (and I hesitate to call anyone using this a learner) to print out lists and match them up with a pen, or require a series of simple “either-or” choices that are too simple to be useful.

"Interactive" pen and paper
CLICK image or link to enlarge

This course, then, is basically just a 152-slide PowerPoint presentation. There is nothing wrong with putting PowerPoint presentations online, if they are creatively put together. A good PowerPoint can be engaging –- applying the limited but still useful possibilities for animation that PowerPoint has, so that ideas are sequenced and highlighted.

Interesting substantive points about data collection and reporting

The sections on data collection (section 5) which runs from slides 101-126 and reporting results using the Story Pyramid, (which runs roughly from slide 138-146) actually have some useful material, but both could have been much better if this material was presented in a more engaging and interactive manner.

The slides on qualitative and quantitative data, and causality have some interesting ideas, for example.

But even here, what we see in this screenshot is exactly what we get – the whole page is dropped in front of the viewer at one load, no drop-downs, no sequencing, no animation, no emphasis – essentially just what we would get, with much less trouble, if we read one of the many PDF manuals or Word documents on results-based management available from UN sources.

Discussing qualitative and quantitative data collection - static format
CLICK image or link to enlarge

A substantial amount of the text in these screens is made from photographs of text – jpg files inserted into the slide. Pasting text like this is a bad idea even for PowerPoint presenations where viewers have a big screen, and a facilitator to wake them up with a joke or two – but for a static, online view, it makes reading difficult.

Jargon and out of date RBM terms

This course slips into academic jargon from time to time, and many of the 152 slides have long text. References to the “post hoc ergo prompter hoc fallacy” and the “dichotomy between quantitative and qualitative research” may show the erudition of the writers, but do nothing for intercultural adult learning.

Another potential problem is the outdated UN RBM jargon. Take the definition of "Outputs" for example:

Outdated results definitions
CLICK image or link to enlarge

For a number of years the UN has been moving very slowly towards definitions of Outputs which see them more as a result – a real change in capacity - rather than just the completed activities that so many field-level projects appear to settle for in their reporting. The current (as of 2011) UN definition of "Output", for example, from the UNDG RBM Handbook looks like this:

“Outputs are changes in skills or abilities, or the availability of new products and services that must be achieved with the resources provided and within the time-period specified. Outputs are the level of result where the clear comparative advantages of individual agencies emerge and where accountability is clearest. As stated in section 1.7, UNDAF results should be formulated in change language.” [p.13]

Even the 2007 UNDG technical brief on Outputs , written within a year of the first apparent appearance of the UN office of Internal Oversight Services online RBM course, had a more complete view how Outputs should be viewed in reporting:

"You may be tempted to list things like workshops and seminars as outputs. After all, they are deliverable and some workshops can be strategic if they gather decision takers in one room to build consensus. But, in most cases, workshops and seminars are activities rather than outputs. And remember that outputs are not completed activities – they are the tangible changes in products and services, new skills or abilities that result from the completion of several activities. " [p.2 - italics added]

The same statement is included in the 2011 updated Technical Briefs on Outcomes, Outputs, Indicators, and Risks & Assumptionsn Outcomes [p. 5], so the idea of accountability for genuine results is not disappearing.

But while some in the UN are conscientiously trying to move staff to reporting on real results, this programme performance assessment course defines Outputs as essentially a completed activity - and - if people see this definition as current, this will undermine UN attempts to move towards genuine results reporting.

The Black Hole in UN Results Reporting

The problem from the bilateral donors’ point of view with much of the UN results framework is that the ambiguity on whether an Output - the primary level of reporting for individual UN projects - is a completed activity or a genuine change in capacity, has provided unambitious UN project managers with an excuse for easy reporting on completed activities.

Meanwhile, the bilateral donors who actually provide the money for most of these projects, have to report on real results. This UN results-based management course uses the older definition of what an Output is, and because it holds the continued imprimatur of the Office of Internal Oversight Services, it reinforces the laziest side of UN reporting.

This leaves a great black hole in the results chain, with reports on completed activities at the project level, and then a long conceptual leap to country results at the other, with no assessment of the quality or utility of the intervening completed activities. If UN agencies, like their bilateral counterparts, genuinely want to test assumptions about what activities and theories of change actually work - and what does not work, then there is no excuse for not assessing the immediate results -- changes in understanding, attitudes and behaviour and professional practice -- that the activities are intended to produce.

Repeating the most limited interpretations of what is expected in terms of results will do nobody any service if this approach is reinforced by an online course that ranks, at least in January 2012, so high in internet search results.

Similarly, the course course puts substantial attention on “Accomplishments” instead of what the UN agencies have all referred to for several years, as “Outcomes”.

What do you do with an out of date course?

This RBM course is clearly out of date. There is a reference on the last substantive slide (#151), for example, to an exercise using 2002-2003 data, and for potential forthcoming work on a 2004-2005 report.

Using 2003 data
Click image or link to enlarge

The case could be made, I suppose, that the course was originally put online in 2005, so as times change we should not expect too much. But there are multiple yearly search listings for the course, with somebody apparently updating them so they will appear current in search engine results. If you do a date search for "Programme Performance Assessment" on Google for example, you will find listings for this course (as if the course were updated) for 2005, 2006, 2008, and 2011.

Search results for the online course 2005-2011
CLICK on the image to enlarge

I compared the March 2005 version of the course with the 2011 version, and could find, before I dropped off into a semi-comatose state, no difference between them.

It is worth noting that the UN Office of Internal Oversight Services search reference itself only lists this as updated as far as 2008 – although it is unclear to me in comparing them, what - if anything - was changed between 2005 and 2008.

The more recent October 2011 listing is a reference from the UNESCO Opentraining site, which describes the course this way:

“The United Nations has adopted a results-based approach and in this, programme performance assessment is an essential component. It is an invaluable tool for programme managers to achieve the twin goals of accountability and effectiveness. The present training is designed to allow learners to work at their own pace in order to acquire the skills they will need to implement programme performance assessment.”

While the programme performance assessment approach in general may be "an invaluable tool", this course, unfortunately, is not.

Missing links – an indicator of website utility

On the whole, the entire course, while it may, in 2005, have had some potential, is too frustrating to be useful today.

Even where potentially interesting documents are referenced, the links are sometimes dead. For example, section two references a manual on “How to Design and Carry Out Data Collection Strategies for Results-Based Budgeting”. If you have to do results-based budgeting, as some of us do, this could be quite useful – but the link referenced there does not work.

And it is not just that the PDF file may have been moved – that can happen to any of us. The link does not work, because it is actually not written in a valid html format. A critical “dot” has been left out of the html for the link. This looks the same in the 2005 version of the course, as it does in the version online today. But even if you format the link correctly, it is still out of date, and there is, in fact, no downloadable document that I could find, even with considerable time spent on the search.

It is a bit mind-boggling that nobody has noticed this - or any of the other missing links - in the 6 or 7 years this course has been online, and presumably intended for use by people who would want to download the documents. That such broken links exist in the still available 2005, the 2008 and 2011 versions, suggest that in all likelihood nobody has clicked on the missing link in years, and pointed this out to the webmaster. All of us who put work on the web will occasionally find we are left with expired links, and most of the time we try to review our sites every few months, to test the links, and update them. Any competent web analytics programme can tell webmasters when a link is broken, but evidently nobody responsible for this online RBM course much cares about this.

As indicators of website utility go, this one seems to suggest that very few people are actually viewing the course in any detail and, again, that those managing the site also have little interest in whether it works properly or not.

So, if the course is out of date, there are basically three choices here:

a) make it clear, with a little simple editing, that this is a 2005 document, and may be out of date – fair warning to the user – or

b) just take it offline, and

c) replace it, one hopes, with something genuinely interactive or at least up to date.

The Office of Internal Oversight Services website

It should not, perhaps, be surprising to find such outdated material on the UN Office of Internal Oversight Services' website. The UN Internal Evaluation Division section of the website says that

“ IED is committed to providing timely, valid and reliable information from an independent perspective that can be used to strengthen the Organization."

But, while some of OIOS reports are up to date, others are not -- in their own terms -- at all "timely". At least in January 2012, the latest available programme performance report is for the 2006-2007 year

The real indicator for me about how the OIOS looks at updating the site, however, is the section called “What’s new?” When I reviewed this in January 2012, I found that only one of 11 references was to any date after 2009. That one clear reasonably recent link was to a 2011 fraud alert .

Aside from that, here is what the OIOS website considers new:

An announcement for the “launching” of a manual called "Managing for Results: A Guide to Using Evaluation in the Secretariat” – which was produced in 2005;

A 2005 Bulletin from the Secretary-General on whistle blowers;

An undated page on risk management, that has at least one non-functioning link;

An undated Guidance to Programmes for Developing an Evaluation Policy (which the document properties tell us was produced no later than 2008);

A 2008 letter to the New York Times on charges of UN corruption in the Congo;

A 2008 letter to Human Rights Watch responding to allegations about Pakistani and Indian peacekeepers;

A March 2009 Investigations Manual

A March 2009 version of the Internal Audit Division’s Audit Manual;

An advance notice (!) for an upcoming workshop to mark the 15th anniversary of the Office Of Internal Oversight Services – in October 2009;

The latest version of the OIOS information booklet (2009);

2009 Uniform Guidelines for Investigation

Indicators for completed activities - not results

Even where there is more recent material, such as in the Investigations Division monthly performance indicators, the documents are dispiriting if we are looking for any clear guidance on results, not just on completed activities.

There are 26 monthly reports there, ranging from October 2009 – November 2011. Not one of these has indicators about results –

No indicators about the quality of the quality of investigations,
No indicators about sanctions applied to violations,
No indicators about the resolution to investigations
No indicators about what long term changes to which the investigations might have contributed.

All of the indicators refer to completed activities – number of cases received, number processed, age of cases, the kind of things that are interesting indicators of process – but not of results.

So, with this limited type of reporting, and with such a casual approach to keeping materials up to date, it should really be no surprise that this organization's online RBM course should be outdated.

A possible alternative RBM course for UN staff

Now, it may be that this course has been superseded for UN staff, with the online course Measurement for Effective RBM, but if that is the case, it is not at all clear from any of the UN websites.

In any case, for the general public -- those of us working with NGOs, governments or other donors --that RBM course is not accessible. Users must have an "official UN system organization e-mail”, must be nominated (it is not clear who can do this), and there is the small matter of the fee – US$1,850, per person.

That course may well be more interactive and more entertaining than Programme Performance Assessment online course – but for the price, it should be!

The bottom line:

If the creators - or managers - of this online RBM course can’t spare the time to update it, why should you spare the time to read it?

If you want an introduction, that remains valid today, to the processes of thinking about results, I still recommend the University of Wisconsin’s online RBM course -- Enhancing Program Performance with Logic Models. Although it was created 3 years before the UN course, ten years ago, as I write this, it makes much more creative use of what the internet can offer for interactive training, the links work, and it provides, additionally, a complete PDF version of the course as a supplement.

And if you want to match this understanding of the processes of RBM with an understanding of what RBM terms the UN agencies are using, instead of wading through the Programme Performance Assessment in Results-Based Management course, a much better use of your time could go to reviewing the many documents available online at the UNDG RBM site. You will find conflicting information in the documents available there, because the UN itself has evidently not yet sorted itself out on results-based management, but at least you will be getting the latest thinking on the subject from people who are trying to make RBM work in the UN context.

[Edited to update links February 13, 2012]

Online Results-Based Management Training: The University of Wisconsin's Logic Model development course

2011-12-15T12:17:00.006-05:00

-- Greg Armstrong --

The University of Wisconsin’s excellent online interactive Logic Model training is a valuable, easy to use introduction to RBM – and it is free.

Level of Difficulty: Easy to moderate

Primarily Useful as: An introduction to RBM and Logic Models

Length: 3-5 hours if used online, 216 pages in the PDF format

Limitations: Some of the links are out of date

Who this is for

Like other consultants and trainers who work on international development issues, I usually work with groups, either donor or implementing agencies, training 10-200 people on RBM. This reduces the cost (to them) of the training time, travel and other expenses. But readers of this site or the RBM Training website sometimes ask if I can provide training on results based management for individuals. This would, however, be so expensive that it would not be practical. Other options such as enrolling in university courses in Europe or Canada might work, but often require more time, and again, more money, than an individual might be prepared to invest.

But there is online, an excellent resource available to introduce RBM to anyone who has no experience with RBM – or, for that matter, to refresh even the most jaded RBM practitioner’s interest. This is the University of Wisconsin Extension Department’s Enhancing Program Performance with Logic Models. Developed in 2002, and put online in 2003, there is very little in this easy to use and interactive course, that is not still relevant today, in 2011, to those who are trying to understand how to use results-based management.

Format: Interactive Online learning and a downloadable PDF

This course has both an interactive online format, and for those who don’t want to, or cannot use the internet for the 2-5 hours completion of the course takes, a PDF version of the course. The real charm of this course, however, is that it is interactive, and it is the engaging nature of the interaction that cannot be replicated in the downloadable PDF. Presentations, in the course’s 7 sections, both audio and text, are supplemented with pop-up windows which participants can view, if they want additional examples or references, followed with exercises and then feedback on the answers.

University of Wisconsin Extension Online Logic Model Training
Copyright: 2002 Board of Regents, University of Wisconsin

It should not be surprising that the interactive elements are as engaging as I found them to be, because this site was developed by a team of content and technical experts, Ellen Taylor-Powell, Larry Jones and Ellen Henert at the Extension Department of the University of Wisconsin – and extension departments are almost always the group in any university the most skilled in tailoring learning events to learners’ needs and learning styles.

Each of the course’s 7 sections contains the following elements:

A Section overview page, which includes

Introductory audio
Section learning objectives
Printable section outline to help users track their progress
Content presentation
Activities that require us to put into practice the theory on the preceding pages.

A discussion of some technical limitations, and the reasonably good alternatives the course uses, are included at the end of this post.

Course Contents: Reinforcing important lessons about RBM

The course title is Enhancing Program Performance with Logic Models, but it deals with all of the associated elements of results-based management, not just the Logic Model. Nothing in it will be a major surprise to people who work regularly with RBM, but the what this course teaches users about results-based management is worth repeating, simply because these lessons are often neglected or ignored, by even the most experienced users of RBM.

The course has 7 content sections, each of which generally takes between 20-40 minutes to complete.

What is a Logic Model?

Section 1 discusses the difference between Outputs and Outcomes, the need to test assumptions, and identify risk. This section covers 20 screens in the course, and runs from pages 7-58 in the downloadable PDF. The difference is that in the PDF, transcripts of the audio from the online course, and a number of worksheets, are included as text.

This introduction to Logic Models includes, among much else, an interesting interactive Logic Model puzzle, requiring us to test our understanding of sequence, timing, risk and results, placing 20 different statement about a programme into one of 8 categories such as resources, activities, participants, Outputs, Outcomes, assumptions and external factors (or risk).

University of Wisconsin Logic Model course - interactive puzzle
Copyright: 2002, Board of Regents, University of Wisconsin
CLICK image to enlarge

This is not dissimilar to what you can do in a group with paper and scissors, but nevertheless engaging if you are working alone, and trying to ensure you understand the ideas of sequence, scale, change and other factors.

More about Outcomes

Section 2 takes us in detail through the different types of legitimate results that we can see in at individual, group, agency, system, or community levels, and talks about why participation of stakeholders in defining Outcomes is important.

This section includes material about RBM that more than one of the UN development agencies could usefully review. Some of these agencies – not all, but some important UN agencies - get hung up on completing activities, (Outputs) and never seem to move on, in practice, to assessing whether these lead to any real change, or result (Outcomes).

Section two uses19 screens in the online course, and 32 pages in the PDF, with all of the associated supplementary material which is found in the pop-ups and links, in the online course.

More About Program Logic

Section 3 introduces (or reintroduces to forgetful RBM cognoscenti) the concept of clarifying the logic and assumptions implied in designing activities with the intention of contributing to a result. It covers theories of change, the complexity of real-life programme logic, results chains – what it calls Outcome Chains - and whether it is reasonable to claim causality – part of the current discussion on attribution of results to programme or project interventions.

It is important to note here, for those sometimes justifiably cynical about RBM who focus on complexity, that this course itself makes the point that multiple, often unplanned or unforeseen factors can affect results, not just the programme interventions. It also discusses the idea that a simple, linear logic model may not reveal enough of the factors involved in achieving results, and why more complex logic models may be needed to underpin the simple ones that we are often forced to submit to funding agencies.

Another interactive exercise here, helps us to think through the theory of action behind a programme intervention, and despite the fact that this section has only 9 screens in the online course, and 24 pages in the PDF, the complexity of the interactive exercise on page 7 means it can take well 30 or 40 minutes to finish this section.

University of Wisconsin online Logic Model course, interactive exercise testing theories of action
Copyright: 2002, Board of Regents, University of Wisconsin
CLICK image to enlarge

This exercise is also an example of why, if you can, it is more productive in terms of learning to go to the online course first, before using the PDF, because with the PDF material there is essentially no chance to test your own understanding of the processes, before you see the “answer”.

What does a Logic Model Look Like?

Section 4 discusses why logic models, depending on the situation, purpose and culture involved, can take many different forms. There is no single format for Logic Models that suits all needs when the intellectual process of testing theories of change is involved – even though there may be, for any given donor, only one format they want to see. There are, as the course points out, situations where Logic Models may differ in size and complexity, depending on whether they are being used to test ideas for programme planning, evaluation, communication or for programme and project managers, the core of implementation activities.

Once again, note to complexity theorists, the course makes the point that logic models need not be simplistic, linear creations, but can be useful in helping agencies and individuals understand the complexity of the systems involved in interventions.

This section takes 8 screens in the online course, and 10 pages in the PDF.

How do I draw a Logic Model?

Section 5 is perhaps the most important of the course because it focuses on the real need to develop logic models as a group, not as an individual.

“As you work through this section, you will appreciate that the best way to construct a logic model is with others. While it may be quicker and easier to work alone, try not to. Many people believe that the real value of logic modeling is the PROCESS of creating one and the understanding and consensus that you build about a program as a result.”[Section 5, screen 1, PDF: p. 127]

I can only say I agree completely. In the Results Based Management training sessions I do with donor agencies, implementing organizations and national partners, feedback suggests that 80-90% of participants were absorbed by the Logic Model development process, sometimes surprised at how often the discussions reveal previously unknown or unacknowledged differences in perception among close colleagues, about what the original problem is, what risks and assumptions they have, and what reasonable results could look like.

5 different approaches to Logic Model development

All of the approaches focus on Logic Model development a group. The first four approaches are intended for planning new programmes. These can be found at the online course, by clicking on the links at the bottom of the screen on creating a logic model for a new programme or on page 136 of the PDF. All of these approaches begin with the result, and move back to activities and necessary resources.

Start with the Long-term result or Outcome, and move back through mid-term or intermediate results, to short-term results, then to the Outputs needed to achieve these, then back again to activities and finally to inputs.
Start with the long-term desired result, but then move immediately to activities, which is often the primary interest of participants, and then testing whether these will in fact, contribute to short of mid-term results which can logically relate to the long-term result.
Start with the Long-term result, then brainstorm all of the elements that will affect this – activities, short, mid-term results, participants, risks, then sort them out to see if participants agree on the relationships and sequence.
Juxtapose the situation or problem with a long-term desired change or Outcome then move back through mid-term and short term results necessary to get there, and finally to participants, activities and resources required. This looks similar to the first option, but in fact really does, in practice in groups force those with preconceptions about what the activities should be, to confront the problem clearly, to look for results and then decide what activities and resources are needed.
A fifth approach is also listed separately, on the next screen of the course, and this is for a situation where it is necessary to start logic model development with existing resources and existing activities.

I have seen this happen where there is a second phase to a project, or where there is just too much institutional inertia to reconceptualize how to approach a problem. It is essentially an approach focusing, as I see it, on a search for results to justify what is already being done – or as the this course suggests, where an “off the shelf” programme already exists: Ask about an existing programme why each activity exists, what possible changes it can lead to, and how this can relate to a newly identified problem.

This approach is sometimes necessary, in my experience, when working on RBM with universities, where activities such as degree course work and research are accepted as the core of university activities, and therefore the starting point for interventions. It can also be necessary with some government agencies which also sometimes see every problem through the paradigm of their own mandate and existing expertise.

Moving such institutions towards a genuine questioning of what is really likely to achieve results is sometimes quite difficult.

How Good is my Logic Model?

Section 6 reviews some of the pitfalls individuals or RBM workshop facilitators may encounter in the development of Logic Models, including:

Getting lost in the RBM terminology,
Focusing too much on the mechanical aspects of putting activities and results in boxes, without assessing the plausibility of the connections between activities and results,
Focusing on – or complaining about – linearity, rather than exploring the complexities that some Logic Model formats can reveal,
Confusing the development of a Logic Model with evaluation, (for which it can indeed be a useful tool, but to which its utility is not limited)
Perceiving the Logic Model as a panacea for programme or project design or implementation problems, rather than as a tool to help us find possible solutions to such problems,
Focusing on production of the paper product, but never using it in practice.

It also makes the point that different people and agencies can use – or require the use of – Logic Models, for different purposes, and that we need to be clear about what this purpose is when we determine in how much detail we will work on Logic Model development.

This section spans 10 screens in the online programme, and 16 pages in the PDF version.

Using Logic Models in Evaluation - Indicators and Measures

Section 7 reviews how understanding the original problem, assumptions, risk and the intended logic of the results chain can help agencies determine what evaluation questions to ask during different types of evaluations – questions about, and indicators relevant to:

The quality of inputs and completed activities (Outputs),
Who is participating, and what is the reach of the activities,
Assumptions underlying programme design, and selection of activities,
Whether results are actually achieved, and to what extent they can be reasonably attributed to the intervention, and
External factors –including what many agencies refer to as risk - on the achievement of results, or the failure to achieve results.

Examination of a Logic Model and how it is formed can provide the foundation for

Needs assessments (assessing the original problem, and what can be done about it),
Process evaluations (assessment of inputs and the quality of activities)
Outcome and Impact evaluations (whether changes occurred and to what extent programme or project activities, and also external factors, may have contributed to such change).

Section 7 is the longest of the course, with 20 online screens and 49 pages in the downloadable PDF. The indicator discussions run from screens 9-20 in section 7, and pages 178-205 in the PDF.

Glossaries and references on logic model development, RBM and evaluation

The course, formally ends with section 7, but it is worth taking a look at other resources included:

The useful glossary of terms related to results-based management, evaluation, data collection and data analysis. Plain language definitons for 61 terms are provided, and this can be accessed on every page of the online course from the upper right side of each screen, or from the PDF on pages 207-211.

The bibliography of references related to results based management and evaluation can also be reached by clicking on “resources” at the top of each screen or on pages 212-216 of the PDF. The bibliography includes 72 references, 8 of which have clickable links that still appear to be functional. Most of the 72 articles or references were written between 1994-2003, but it is worthwhile in particular visiting the Centers for Disease Control evaluation resources page which has a lot of very useful, accessible, and in some cases more current, guides on evaluation, logic models and data collection. The link in the bibliography is not current, but the page will automatically redirect to the new location, which I have provided in the previous sentence.

11 additional links on evaluation issues such as questionnaire design, surveys, focus groups, quasi-experimental design and other issues can be found throughout section 7 online, or on page199 of the PDF.

Finally, the course also provides links to 15 downloadable logic model worksheets or hints, in PDF and sometimes Microsoft Word format, also under “resources” at the top of each page. These can also be found scattered throughout the PDF document.

Limitations - Technical issues:

Because this course was developed in 2002, most people will have computers capable of making use of the interactive elements of the site – but there are three potential limitations to this, all of which, however, are dealt with in some way by the designers:

1. A reliable internet connection is needed to use the site, and in many of the places I, and my colleagues have work such connections can be, at times, unreliable. You might, for example, have a half hour of access to read things like this blog, but it would be frustrating to be interrupted in the middle of this online course.

Alternative provided: Users can download the 216 page PDF version of the course, which includes the text of the whole course.

2. Flash is used for the most compelling of the interactive features, such as drag and drop creation of logic models – and this, as I understand it, is unlikely to work for people using Apple products. Most of the people I work with used Windows-based computers, and have the flash player installed, so this might not be a major problem, but the flash has to be enabled, and some people, and some network administrators, do disable it for security purposes. The website provides a link to the free download of the flash player, from Adobe. Similarly popup screens and forms provide a wealth of additional detail in every section, and some web browsers may require users to enable these through security settings.

Alternative provided: The creators of the site provide an alternative to the use of flash, with some interactive elements, using links. It is not as compelling from a learning point of view as using the flash elements, but it does permit some interaction.

3. The audio portions of the presentation use the “.ram” audio format, and a download of either RealPlayer or one of the alternatives such as VLC player is required to listen.

Alternative provided: When I first came across this course, in 2008, there was, essentially just the online version but in 2010 the PDF version was produced and it is a useful reference. Some of the links in the document are out of date, but many more of them still work, and they are themselves quite useful.

All of these, and some other technical issues are addressed also in the course online help link and the Extension department provides an email link for those who may have further questions about the content or format.

The bottom line: This is an excellent introduction to results based management, focusing on logic model development, and its design is a credit to its authors' adult education abilities. It does not pretend to replace group workshops but it provides an intelligent, practical, and easy to use walk-through of the main issues in RBM. PDF technology has moved on since 2002 and it is possible to use both flash and javascript now in PDF files. It would be interesting to see what the digital media specialists at the University of Wisconsin could do with the PDF version of this course, to introduce some of the dynamic elements of the online course.

Podcasts 3: BBC's More or Less: Behind the Stats

2011-08-02T00:51:00.010-04:00

Greg Armstrong --

This BBC Podcast brings indicator discussions to life. BBC radio's weekly podcast More or Less: Behind the Stats presents an entertaining array of examples of how intelligent people can differ in their interpretations of indicator data, and how to apply common sense to the claims made using statistics, or other quantitative data.

BBC Radio's More or Less Podcast

Level of Difficulty: Moderate and entertaining
Primarily useful for: Policy makers, senior managers, Project stakeholders, donors, project managers, project monitors, and politicians of any nationality
Length: 28 minutes (mp3 format)
Limitations: 22 recent programmes are available for download but many older programmes are available only for listening online.

Who this is for

For project managers, donors, project monitors and policy makers who need to maintain a watchful eye on how indicator data are used, More or Less, provides some interesting examples of why it pays to be skeptical of indicator validity and reliability, the claims and the interpretations put on indicator data.

Background: Surveying Results-Relevant Radio

This is the third in a series of posts discussing how audio podcasts can reinvigorate thinking on indicators and results.

The first post in this series discussed the mechanics of downloading and listening to podcasts. The second post surveyed the wide range of available podcasts from the BBC, ABC and National Public Radio of potential use to people working on results and indicators.

This post reviews what I think is the single most useful programme on world radio, for people who work with indicators: BBC's More or Less: Behind the Stats.

BBC Podcasts

BBC is not the only source of intelligent programming available on the internet - there are, as I noted in my June 2011 post, several excellent programmes available from ABC, National Public Radio, and other sources. But BBC radio has by far the widest range of podcasts to choose from.

Roughly 9 Thousand available BBC podcasts

The BBC podcast website the last time I looked (August 2, 2011) had 287 programmes available for download. Of these, roughly half usually fall into categories such as music, comedy, sports, religion or children's programming but 129 programmes fell into the "factual" podcast category. Each of these programmes has multiple -- some dozens, some hundreds -- of individual episodes available for download, or listening online.

The BBC Radio 4 website in early August 2011 listed more than 9,600 individual programme episodes, with almost 9,000 of them still available for listening in some format. This is in the "factual" category alone. Some are BBC news programmes, and despite recent cutbacks to the BBC foreign language programming, news is still available in many languages(see links at the end of this article). News programmes often cease to be available more quickly than other documentaries, for obvious reasons of topicality, but the last time I looked, there were roughly 90 available.

Some of the nearly 9,000 available factual podcasts on radio 4 focus on consumer affairs, arts history or travel.

But there are several which provide useful but also entertaining insights into the kind of work we do when we think about results and how to describe, measure or report on them.

More or Less: Behind the Stats

Of all the programmes I have found online, the most directly and consistently relevant to results-based management is More or Less: Behind the Stats. It has been hosted, since October 2007, by economist Tim Harford, the engagingly skeptical author of The Undercover Economist, The Logic of Life, most recently Adapt and many articles for The Financial Times and other publications. Tim Harford brings a common sense and clear language approach to determining whether claims for results, and the use of statistics to support such claims, are credible.

Respect for Data

In this, he continues the work of his irreverent predecessor, Andrew Dilnot, currently the chair of a U.K. Commission examining long-term health care for the elderly. Andrew Dilnot set the tone for More or Less with his no-nonsense approach to data. As The Guardian recently wrote about the Dilnot report, it is "Rich in evidence and pithy in prose" - and this is, after all, what we need more of, in all of our reporting.

Dilnot wrote a few years ago that what is important in judging political leaders' claims is "respect for data over wishful thinking", something that could be said equally of results claimed for development projects.

"If you'd prefer to be flattered by bogus numbers, to believe that the world changes when you play statistical games, or at least to act as if it does, wrote Dilnot "you are, let's be blunt, delusional and dangerous."

In a world where political leaders exhort aid workers to base -- and justify -- their programming decisions on indicator evidence, but themselves use evidence as the basis for policy only when it suits their political needs, this is something worth remembering.

120 Available episodes of More Or Less

More or Less has been produced since 2005 in association with the Open University and their site has some supplementary written material. Additional interesting material is available also on Tim Harford’s other websites.

The programme is updated for downloads every Friday during its broadcast seasons, and appears to have 2-3 broadcast seasons of 7-9 episodes each year.

The More or Less website had, as of August 2, 2011, 22 episodes available for portable listening in the standard and (easiest to download) format - MP3. (A very rudimentary introduction on how to download and listen to podcasts, is available in my May post.) Each current episode of More or Less is 28 minutes long, and they cover the period between September 2010 and May 2011, the most recent broadcasts. An additional 28 streaming episodes of More or Less are available for listening -- but not downloading -- using the BBC Player, covering the period between January 2009 and August 2010. A new season of More or Less begins in the first week of August 2011, and the September 2010 episodes will probably be archived before the new season ends.

There are also older More or Less episodes, going back to February 2003, with Tim Harford or Andrew Dilnot, but there is no point in trying to access those by clicking on the "previous programmes by year" link, because some work, and some don't. But if you go to the More or Less Archives you can get 73 more episodes. What is curious about these older archived programmes is that some - such as the earliest available episode of More or Less in February 2003, do have audio you can listen to, albeit in the sometimes problematic ram format, while others, including some later programmes, simply have a written description, but do not, as far as I can see, have an audio component. All six of the More or Less episodes broadcast between February and March 2003 have audio, for example, but none of the six episodes broadcast in January and February 2004 appear to have audio.

Nevertheless, all things considered, I estimate that that there are probably about 120 episodes of More or Less that you can listen to one way or the other and as the new season arrives there will be more.

A wide range of indicators

Each episode of More or Less usually deals with discussions of 5-6 indicator issues. Recent episodes, for example, have included discussions on indicators related to, among many other subjects

measuring child poverty
calculating civilian deaths in war zones
comparing international data on student achievement
measuring well-being
abuses of statistical significance claims
measuring the "fiscal multiplier"
how luck, and regression to the mean, can bias data interpretation
whether celebrity (or royal) marriages - or crime - lead to jumps in marriage rates
calculating the real costs of military interventions
how differences in data definitions can bias international comparisons
distinguishing between correlation and causation
how different methods of calculating "averages" can affect indicator data

Using More or Less as a research tool: What the data said about 2010

While many of the items on More or Less use examples from the United Kingdom, it is easy to see how lessons from the debunking of claims about problems or results, could be applied to other issues, and other countries.

A useful starting point for anyone wanting to test the More or Less range of issues, is the December 31, 2010 downloadable episode. The MP3 version of this episode can currently be downloaded from the main More or Less website, with 21 others, but it will soon be archived, and then you will need to go to the BBC player version of the programme, where currently the most recent 48 episodes are available, and listen online.

"The meat and drink of More or Less are the errors and connivances embedded in the statistics which fill each news bulletin" Tim Harford noted in the final programme of 2010. And this episode illustrates his point, as it produces the most important numbers of 2010, as seen by 7 people who work regularly with indicator interpretation.

Examining one episode in slightly more detail provides an example, I hope, of how we can use podcasts to stimulate ideas and, with a little effort, further research on indicators. As there are no clickable links in podcasts (at least not in this one), the references below are to the time (in minutes and seconds) into the podcast where you can find the reference.

Included in the 2010 Year-End summary of indicators in the news:

Indicator data on crime and social change

(00:56)
Daniel Franklin Executive Editor of The Economist, and editor of The World in 2010 and The World in 2011, discusses the difference between David Cameron's claims on crime and a "broken society" and what the statistics on crime rates, teen pregnancy, smoking and other issues say. Some of this is reflected also in the data on percentage of births to teenage mothers, suggested by National Statistician Jil Matheson (08:24-10:19)

It is easy to see how these discussions on falling crime rates and the implication for public policy could apply to other countries, such as Germany the U.S. or Canada

Indicators on defence spending

(02:30 into the episode)

Cathy Newman, former correspondent for the Financial Times, now political correspondent for Britain's Channel 4 news, and author of the Factcheck Blog contrasts former Prime Minister Gordon Brown's claims that defence spending rose in real terms in recent years, while data on inflation - adjusted spending suggested defence spending had fallen. She points out the difference between "cash spending" and "real spending", and she also notes that the Conservative - Liberal Democrat coalition government will be reducing the defence budget by about 8% in real terms over the next four years.

Misleading indicators on immigration

(04:25)

Tim Harford highlights the need to check indicator data sources, before making extravagant claims. He shows how Liberal Democratic party leader Nick Clegg's election debate claim that 80% of immigrants to the UK came from the European Union, was based on his party's misreading of an Economist article, which referred to students, not immigrants. The real figure Harford says, and a report in the Daily Telegraph appears to confirm, is about 39%. It is not known from any of these sources if the use of the faulty data was sloppy party research or careless use of the data in the debate.

Compared to what? Risk indicators

(07:30)

David Spiegelhalter, The Winton Professor of the Public Understanding of Risk at Cambridge University, talks about the need to put risk indicators in perspective. He examines risks for the military in Afghanistan, and compares these to the risk of riding a motorcycle on a major highway in the U.K.

Hans Rosling on the quality of indicator data

(10:30-18:10)

In the longest discussion in this episode, Hans Rosling, co-founder of the Gapminder Institute, who will be familiar to many people from his entertaining presentations on indicators , talks about how indicator data on issues such as child mortality and economic growth differ widely country to country in sub-Saharan Africa.

When questioned about the reliability of indicator data from under-funded African statistical offices, he explains why some indicator data such as child mortality and fertility rates are reliable, while other data on indicators such as maternal mortality and unemployment are not.

But, Hans Rosling says:

"It is not countries that have weak indicators. It is certain indicators that are weak for methodological reasons". (13:23)

There is much more in this interview with Rosling, on:

Indicators on fertility;
Rates of change to indicators such as child mortality, and economic growth in China;
Disaggregating indicator data and defining geographic focus for indicators;
His conclusions about the relationship between economic improvement, good governance and democracy;
The difference between ideals and advocacy, and facts.

Finding an indicator for the "slippery concept" of well-being
(18:37-20:24)

Statistician Michael Blastland former producer of More or Less, co-author with former host Andrew Dilnot of The Numbers Game: The Commonsense Guide to Understanding Numbers in the News, in Politics, and in Life , and author of a regularly published column on the BBC News online Magazine on making statistics relevant to non-statisticians, considers whether insomnia can be taken as an indicator for wellbeing - and concludes "...Well-being: this is a good illustration of how slippery a concept that is and the number of things that might have to go into it".

Dealing with slippery concepts is something development aid workers will be familiar with.

Cash as an indicator of bank viability

(20:26)

Robert Peston, Business Editor at the BBC talks about how the amount of cash banks have on hand as a percentage of what they borrowed, is an indicator of possible bank failures. Comparing British banks at the end of 2008, with British banks during the Great Depression, he comes up with some surprising information. It is presumably still relevant in 2011, but the interview never quite makes clear how

How Incomplete Data reporting undermines indicator utility

(22:50)

Ben Goldacre, a physician, the author of the Bad Science blog, a book of the same name, and a columnist for the Guardian uses the case of incompletely reported data on the drug reboxetine to illustrate how published studies on drug effectiveness provide unreliable indicators for safety and effectiveness because of the way data are both withheld, and then reported. While this brief comment focuses on this one drug, his blog has discussed several other examples of how inaccurately reported data on drug trials can be, to put it mildly, unreliable. These include, within the past year, reviews of a medicine for schizophrenia that may cause diabetes, a diabetes medication that may increase risks of heart attacks and several critiques on claims for homeopathic medicine.

This short session of More or Less with Ben Goldacre is of relevance for those involved in results reporting for any field. Checking the methods and context for any research providing us with indicator data, is something that is often neglected as indicators are used in reports on development projects. Failing to check out the sources for published results supporting results indicators in development projects, however, rarely has such immediately dangerous implications as does careless - or malevolent - use of indicators in the field of medical research.

[Editorial note: February 2012: For an entertaining infographic on Bad Science see the Bad Science Infographic.]

Limitations:

Unbundling....other programmes do it:

As I noted in my previous survey of a number of results-relevant podcasts, several -- such as ABC's Counterpoint unbundle the programmes -- break them down into components that can be downloaded separately. ABC Radio's The Science Show makes the case for this when it says:

"The whole program cut up into separate stories - allows easy skipping from one story to the next so you can pick and choose".

What is also useful is that the Science show also makes transcripts available for many episdoes, so having listened to it, it is relatively easy to go back, check data and follow up.

Some (but not all) episodes of the National Public Radio show RadioLab also do this quite well, dividing an episode which has its own internal coherence, into 3-4 components available separately and with their own references. You can see examples of these, among many others, with the May 31, 2011 episode on talking to machines or the June 2009 episode on randomness and data patterns (stochasticity). There may be some BBC programmes that do unbundle their podcasts by themes, but I haven't found them yet.

Time limits:

While the 22 recent episodes (this will undoubtedly change in August 2011 with the new season) are available for download and portable listening, and readers can listen to, but not download, many others dating back to April 2003, some of the programmes prior to 2005 do not open easily. The easiest to download format - the MP3 versions currently available for the last 22 episodes - are usually only available in that format for a year. So if you want to be sure you can download and save some of the interesting episodes, it is worth skimming the site first, identifying potentially interesting episodes, and downloading them before they are archived or go to the BBC Player site. The September 2010 episodes will probably be archived soon.

Weak research links:

More or Less is relevant to those who work with results and indicators, but of course it provides just tantalizing summaries of the issues, not the whole picture. It is natural that many of us would want to do further research online to follow up on the issue summaries in each episode. This is the whole point of being online - to get access to multiple sources of data. We can, of course, do our own research, and many of the links I provided earlier in this post when I discussed the 2010 New Year's episode of More or Less, were links I did find myself, through some time-consuming research.

While the More or Less website has improved since January 2011, and particularly since the April 2011 season, it is still a disappointment that there is so little assistance on the site for further research, particularly for some of the programmes that predate 2011. The More or Less website does indeed now provide more links that it previously did, both directly and through the Open University link, but there remain occasional problems even in these links. In at least one case I noted in January 2010 the site misspelled the name of one of the guests (Ben Goldacre) and that error remains as I write this in July. Such things prove distracting, and complicate attempts to follow the story online with further search. While it is easy to make errors in writing (and this blog undoubtedly has some) there is a certain irony in finding such an error on the website of a podcast dealing with data integrity.

Other podcasts such as RadioLab have websites that provide much more in terms of links to further research, and in RadioLab's case, even have a reading list. None of these other programmes has the direct and frequent relevance to results and indicators that More or Less has, but they are better organized.

More or Less is, as I have said, the most useful of all of the programmes I have found so far, to those of us who work with results and indicators, and it is disappointing that the website does not make it easier to capitalize on the good work they have done. More links to background material on the More or Less website would be useful as would even the inclusion of metadata accessible to listeners who might want to go further.

The Bottom Line:

More or Less: Behind the Stats, is the most useful programme on the internet for people interested in how indicators are used in daily life, in policy analysis, politics and assessing results. It is worth listening to, for most people involved with results-based management. A new season of More or Less begins on August 5, 2011.

Podcasts and RBM - 2: Audio Podcasts from the BBC, Australian Broadcasting Corporation and National Public Radio

2011-06-30T00:12:00.012-04:00

Greg Armstrong --

Radio programmes from the BBC, the Australian Broadcasting Corporation and National Public Radio provide stimulating insight into how other people work with and interpret indicators and results. The second of three posts on podcasts surveys the programmes available from these three broadcasters that are relevant to Results-Based Management discussions.

Level of Difficulty: Moderate-complex, but entertaining
Primarily useful for: Anyone who wants or needs fresh insights on results and indicators
Length: Usually Vary from 15 minutes to an hour. (mp3 format)
Limitations: Audio podcasts are difficult to reference, and follow up, compared to other media.

Who these programmes are for:

Some project managers dread discussions of indicator data, possibly because they rarely collect the data. But many project stakeholders, and those managers who do take the process of indicator development and data collection seriously, often seem energized by the discussions. Indicator development discussions reveal the different priorities stakeholders have, and what they think results really mean; and they challenge participants to think critically and creatively about data sources, data validity and the practicality of data collection. For these people, audio podcasts -- available free on the internet -- can provide though-provoking insights into both results and indicator development.

This is the second of three posts on audio podcasts. The first post, originally published May 28, 2011 dealt with the advantages, disadvantages, and mechanics of downloading, listening to, and using podcasts for RBM. This post surveys results-relevant podcasts available from the BBC, Radio Australia, and National Public Radio. The final post will review one programme, BBC’s More or Less, in more detail.

Thousands of podcasts to choose from

There are hundreds of possible programmes, and thousands of individual episodes of programmes available for listening, free on the internet. The BBC website alone had 262 separate available programmes, in January 2011, and by June had 288 programmes with material available for download. Of these, roughly were in categories such as music, comedy, sports, religion or children's programming. 122 programmes fell into BBC's "factual" category in January 2011. By June 2011 there were 133 factual programmes listed and most of these had dozens, sometimes hundreds of individual episodes available either for download or listening online -- some news programmes, in multiple languages, others about consumer affairs, arts, literature, economics or history.

The BBC radio 4 website itself suggests that there are over 9,000 episodes of different programmes just in this “factual” category available for listening in one of its formats, listing them alphabetically and by genre. Compared to the BBC podcast homepage, which organizes the available programmes into more recognizable categories, the 9,000 available podcasts may seem like a huge and unfathomable number to wade through -- but these radio 4 episodes are worth skimming. Some individual episodes buried there -- such the interesting 2008 Peer Review in the Dock -- do not appear to be listed on the podcast page.

But this blog is about Results-Based Management and, given that we all have limited time available for listening, the following are suggestions of some of the programmes I think are worth listening to for useful -- but also entertaining -- insights into the kind of work we do when we think about results and how to describe, measure or report on them:

Results-relevant Podcasts from the BBC

BBC's More or Less: Behind the Stats, is by far the programme with the most direct link to indicators and results based management, of any I have found. Each 24-minute episode usually includes 3-4 issues, all of which are directly relevant to how results and indicators can be interpreted. 22 individual episodes dating back to September 2010 are available for download as I write this, and 82 more going back to 2005 are available for listening online. I will review More or Less in more detail in my next post.

Thinking Allowed, a half-hour programme focusing on social science research, currently has a total of 228 episodes available -- 40 in downloadable MP3 format, dating back to September 2010. The Thinking Allowed Archives includes broadcasts going back as far as January 2007, using the BBC iPlayer.

Documentaries is by far the most prolifically accessible of all of the BBC podcasts. It had 88 24- minute episodes available, all downloadable, at the end of June -- and this is just for 2011. Another 660 downloadable programmes in MP3 format in the archive from 2007-2010. Finding these archived materials is not perfectly intuitive, but you can get access to them by going to the BBC factual/history link where, among many other programmes, the Documentaries for 2007, 2008, 2009 and 2010 are listed.

Material World, a BBC science programme had 41 half-hour episodes available for downloading the last time I looked, and 350 more in the archives for which the listener will require either RealPlayer or another media player such as VLC, capable of handling the Realplayer files.

Results-relevant Podcasts from the Australian Broadcasting Corporation (ABC Radio)

ABC radio’s Counterpoint, which is second only to BBC’s More or Less in its relevance to results discussions, delivers weekly one-hour programmes, and also "unbundles" the components – breaks the programme up into shorter segments which can be downloaded or listened to individually. Thus, you might want to listen to just that part of the Counterpoint February 14, 2011 broadcast on the “decline effect” -- or why much apparently validated published research can’t be trusted – but not those parts of the same broadcast dealing with Australian politics, limits to online publishing freedom, or the morality of long-term debt

Hiding the pigs…

The only quibble I have with Counterpoint’s unbundling

For example, the June 6, 2011 episode of Counterpoint included three components: “You’ve got to be rich to work for free”, “Hunters, the real conservationists” and “David Burchell: Anger, politics and the new media”. Looking at these, you might not, (and I did not) expect that one dealt with the fascinating issue of how Australians are trying to deal with 23 million highly intelligent feral pigs that are roaming the country. I’m not sure this has anything to do with RBM, but it’s interesting! When I pointed this out to my colleague Anne Bernard, who first led me to Counterpoint, and someone who listens to every episode in its entirety, her reply was “Armstrong, you have the attention span of a gnat! Just download the whole programme!”

But, if, like me, your attention span argues against downloading an hour of material just to find out about the pigs, you can, as I did, just download the feral pig episode

And if the pigs don't interest you, there are a number of other topics of potential relevance for results and indicator discussions in recent available podcasts of Counterpoint:

The two-part September 27, 2010 and October 4 2010 segments of Counterpoint questioning the reliability of expert opinion in general, and examining the particular problems of validity of case study data used in much popular management research.

The segment of the October 18, 2010 Counterpoint episode discussing levels of confidence in scientific understanding on environmental indicators.

The December 6, 2010 episode dealing with the relationship between expenditures on education and educational outcomes.

The segment of the April 18, 2011 broadcast on the dangers, and “unfortunate unintended consequences” of trying to turn qualitative assessments of excellence, into quantitative rankings.

The June 20, 2011 broadcast which contains two interesting indicator-related segments: one on how the quality of data collection instruments can affect data quality, and with it our conclusions about results; and another on what data tell us about the relative contributions to project results and organizational success made by senior management, mid-level project managers, and creative personnel.

Ockham’s Razor is another ABC programme which takes a slightly more academic approach to issues, with individuals making presentations on simple truths behind complex issues, rather than being interviewed. The programme has 240 13-minute episodes going back to January 2006 available for download, and a large number of transcripts for programmes as far back as 1997. These include, among much else, discussion of the difficulties of relating and working with a bizarre field of indicators for earthquake prediction, how simple language and basic math can bring policy debates into perspective, and why effective and simple solutions to policy issues are not implemented.

As an example of how many episodes are available, and on what variety of topics, a search for "evidence" in the Ockham’s Razor archives, produces a list of several hundred presentations.

A Results-Relevant Programme from National Public Radio

Radiolab – a programme from the U.S. National Public Radio produces a one hour episode every two weeks, and these can be downloaded as one entire episode, or you can choose, as with ABC’s Counterpoint, to download component parts of the episode, lasting10-30 minutes each. The format is much more story-telling than "More or Less" or "Counterpoint", and while dramatic liberties may sometimes be taken with the narratives, there is a lot of interesting material here. In total, by the end of June 2011, there were 46 one-hour episodes available. Not all of them are obviously related to results or indicators, but they are all worthy of attention. Two of the episodes I found particularly interesting were:

The June 2010 Radiolab episode “Oops” – which tells three stories about unintended and very negative results, growing out of projects with only good intentions, and
The October 2010 Radiolab episode on "Cities" particularly the component called “It’s Alive?” that describes how speed (of talking, and walking) can be used as an indicator of city culture and how physicists have used walking speed to predict city size, average income, crime rates and a number of other variables related to the culture of different cities.

The bottom line:

BBC’s "More or Less" and ABC’s "Counterpoint" provide a good starting point for anyone wanting a little entertainment with their results and indicator discussions. There are dozens of other programmes out there that I haven’t covered, and no doubt many more that I am not even aware of. Many of these may be of interest to you, and may also, as an incidental byproduct of your attention, provide new ways of looking at results and indicators.

Further listening

Referenced here:

BBC: More or Less: Behind the Stats
BBC: Thinking Allowed
BBC: Documentaries
BBC: Material World
ABC: Counterpoint
ABC: Ockham’s Razor
NPR: Radiolab
NPR: Krulwich Wonders

Other radio programmes of potential interest include:

BBC: The Reith Lecture Archives
BBC: Start the Week, with Andrew Marr
BBC: Four Thought
BBC: File on Four
Podcasts from the Guardian
Podcasts from the Scientific American

This post edited July 4, 2011 to update links.

Podcasts and RBM 1: How to use audio podcasts to reinvigorate thinking on indicators and results

2011-05-30T16:43:00.199-04:00

Greg Armstrong --

Indicator discussions don’t have to be boring. A wide range of audio podcasts, easily accessible to listeners throughout the world, are available online from the BBC, ABC and NPR. This is the first of three posts surveying audio programmes available online, and relevant to results-based management.

Level of Difficulty: Moderate-complex, but entertaining
Primarily useful for: People who don’t have experience with downloading podcasts
Length: Usually Vary from 15 minutes to an hour. (mp3 format)
Limitations: Audio podcasts are difficult to reference, and follow up, compared to other media.

Who this post is for:

This is the first of three posts dealing with how, and which audio podcasts can be useful for people working on results frameworks and indicators. This introduction explains why podcasts can be useful, what their limitations are, and how to use them. This post is intended primarily for people who do not already know how to get access to, or use podcasts.

The second post in this series will survey the broad range of podcasts available primarily from the BBC, Radio Australia and one from the U.S. National Public Radio. The third post will review in more detail one particular programme on the BBC, More or Less which always has something useful to say about indicators.

Those readers who already know how to subscribe to, or download podcasts may wish to skip this post, and move on to the next two.

Why use audio podcasts in RBM? Because (gasp!) RBM can be boring

All of us who work with international development projects or with Results-Based Management, are familiar with the reams of paper, log frames, risk management frameworks and charts, generated in results and indicator discussions. These can put even the most enthusiastic proponents of results-based management into a coma.

But there is a range of very entertaining material available for listening, that can reinvigorate interest in how results and indicator data can be manipulated, misrepresented, forged, and in some inspiring instances, creatively interpreted --and not just in politics, or development assistance, but in daily life. For this, the audio podcasts available for download, or for online listening from a number of sources, are a useful and energizing source of not just learning, but entertainment, for people who work with results and indicator data on a regular basis.

Podcasts provide an escape from the drudgery of reading about RBM

I am a late adopter, someone slow to embrace new technologies, after an early, expensive and ultimately futile early adoption of the Betamax in the 1975. While my closest colleague has for many years been downloading to her MP3 player and listening to not just music but documentaries, and fiction, I only grudgingly started to do so a few months ago, when I began an exercise regime that put me in a boring environment for an hour a day. Music doesn’t provide the escape for me it does for many people, and I wanted to use the time productively. Early misguided attempts at reading while exercising produced unintended (but in hindsight predictably disastrous) results.

My colleague pointed me to the BBC website and its literally hundreds of podcasts; I continue to use it, but also moved on from there to the Australian Broadcasting Corporation’s smaller but worthwhile set of documentaries and then to the National Public Radio site. Now I find that an hour of exercise is intellectually productive and, best of all, entertaining. These podcasts provide insights on how other people deal with results and indicators in the real world, challenging my understanding of issues, and ways of thinking about them, and providing me with alternative approaches to data analysis, often things that I had skipped over, or forgotten in my daily reading. Some of these programmes are engrossing enough that I double my exercise time to complete them.

The drama of results and indicator discussions

The major attraction of using audio podcasts, for me, is this entertainment value. It is rare that a discussion, even on apparently boring topics related to results or indicators will make its way to an audio podcast on any of the major radio networks, unless there is an interesting or unusual twist to it. These programmes are often presented in a way that will stimulate the listener intellectually or emotionally, sometimes reawakening a dying interest in how to use data productively.

Listening to politicians or pharmaceutical manufacturers twist data, and then face the challenge of someone who knows enough to ask pointed and challenging questions, is much more interesting than reading the same discussion in a journal, a newspaper or online. Debates on issues such as health services, school quality, risk, crime, disastrously unintended results, and a wide range of other topics, can generate new ideas for people working with results frameworks, and struggling to recognize, generate or interpret convincing indicator data.

Some programmes such as WNYC’s Radio Lab deliberately dramatise the discussions to keep listeners involved, and that approach is effective. But most programmes benefit simply from focused questions, good editing, and the energy and passion of the people they are interviewing, to keep a listener’s interest. Many of them remind me of the best indicator discussions in a project context, when stakeholders understand the importance of indicators for defining results and activities that are important to them, and look forward to and passionately engage in the discussions about what they mean.

Podcast length

Most of the podcasts I listen to are about 30 minutes long. Some programmes such as ABC’s Counterpoint are an hour long, but listeners can download individual components of the episode, that might vary from 10 to 30 minutes in length. Some sites, such as The Scientific American /, have podcasts that last only a minute or two while others last roughly 15 minutes. ,

Difficulties in referencing or sharing podcast data

The primary disadvantage of using podcasts as a source of new ideas is that it is very difficult to footnote or bookmark the programmes. Only a few podcasts provide transcripts of their audio programmes and among those which do, such as ABC’s Ockham’s Razor , even fewer make use of the primary advantage of the internet – web links.

With paper, we can footnote references, drawing attention to individual words, sentences or ideas, and move back through an article to check consistency or the spelling of a name or an organization.

With electronic data, available on websites, we can provide links, from a blog such as this, so readers can jump to original or alternative sources, to document or challenge an idea, and readers can easily supplement ideas by using search engines.

But if you download a podcast, and you find a startling new idea you want to reference, while you are, for example, jogging, climbing stairs, lifting weights, or walking, how do you do it? I tried carrying a notebook and jotting down the ideas, but this is distracting and sometimes dangerous if you are exercising.
And it doesn’t work well in the rain.

In these cases the only useful way to actually use the podcast as a source of potential learning and a reference for other people, (at least as far as I know) is to listen to it on a computer, then go to the podcast home page, to note the web links to the individual podcast, and sometimes to note the running time of the particular quote within the podcast. Then, too, we can check the website’s home page and links for supplementary information.

So, while I often start now listening, for example, to BBC’s excellent More or Less as I exercise, I often end up listening to it again, in front of a computer, where I can pause the programme, make notes, rewind, or fast forward to relevant sections of the discussion – or jump to the web where I can seek supplementary information. I will be reviewing More or Less in greater detail in my third post in this series, and the difficulty in referencing individual stories in a programme will be illustrated more clearly in that post.

In any case, I assume everyone who reads this blog will have access to a computer – so it should be possible to go directly to some of the sites and programmes I list in my next post, and listen to them online.

The Mechanics of accessing and listening to podcasts

The most common format for podcasts is MP3. While not providing great sound for music (so I am told) this format is, certainly to my impaired hearing, good enough to deliver an audible conversation, debate or discussion. MP3 players such as the iPod all include software to play podcasts automatically. You can spend a lot of money on MP3 players if you want to, but there are perfectly serviceable models, such as the one I use, available in most countries for roughly $20 U.S. And if you decide you want to listen to these on the computer, any reasonably modern computer with a media player, such as the ubiquitous Windows Media Player, Apple’s ITunes or one of the many free alternative media players, will automatically start playing these programmes once you click on them.

There are also other formats, sometimes proprietary, used by individual websites. BBC, for example, while making podcasts for almost all of its programmes available in MP3 format when they are first put on the site, has a few that can only be listened to with its BBC iPlayer, online. This requires an updated flash player and I have had uneven success in using this where internet connections are slow. Some older archived BBC episodes from 2005 or earlier, may only be available in the ram format, which requires RealPlayer or an alternative, and these will still start automatically on most computers when you click on the file. A few of these BBC radio programmes, primarily music, are restricted in places like Canada, by the BBC licensing of its products.

Listeners can also subscribe to audio programmes through links on the site, or through aggregators such as iTunes, or Google Reader, or directly through links on the podcast webpage. Episodes can then be automatically delivered to the computer or mp3 player. Personally, I prefer to select the individual programmes, read the background, and download them myself, but many people prefer the convenience of automatic delivery.

There are, then, as far as I know, two primary ways of listening to some of this excellent material

a) With earphones, downloaded to an MP3 player or smartphone, or
b) Through your computer, by clicking the appropriate link.

Most websites can use all of the major web browsers, but there are sometimes minor differences in how you download a podcast using Google Chrome, or for example, Internet Explorer, Opera, Firefox or Safari. Most of the time, on most of the sites, the link to the specific audio programme or episode you want will give you fairly straightforward instructions, to either click to download, or click to listen at your computer, and where it doesn’t do this, left clicking will usually lead to the programme playing immediately on your computer while right clicking will often download the whole programme to your computer, for listening later either on the computer or on the MP3 player.

Time limits on available programmes

Websites vary widely in how long they will keep an individual podcast publicly available. Some do it as a matter of policy for three months, others for a month, some only for a week and some, such as NPR’s Krulwich Wonders -- basically a written blog -- appear to have only monthly episodes in audio, but many more episodes as written blog posts that can be read later.

In the case of the BBC podcasts, the length of time differs depending on the programme. Some are available for download for years, others for only a week, after which they may completely disappear, or be available only for immediate listening at your computer, but not for download.

As the BBC’s podcast website help page explains it

“But please don't forget that once you have downloaded a podcast episode, it is yours to keep forever and will not expire. Unfortunately, if you missed an episode and didn't download it within the period of availability we are not able to send you a copy.”

So, if you find something that is only potentially interesting, it is worthwhile downloading it for further review first to a computer, and then moving it, if you want, to an MP3 player, or simply keeping it for later listening. The files, because they are relatively low-fidelity for conversation, do not take up as much space on the computer as would higher fidelity music files.

What topics are available?

In my next post I will provide an overview of some of the most interesting radio programmes available, that are relevant to Results-Based Management.

The bottom line:

Podcasts can be difficult to work with as references, but they are stimulating additions to the dense written material we work with regularly, and can be a useful additional tool for people who think about and work with results and indicators.

Further reading on how to listen to podcasts:

BBC podcast help
ABC podcast help
Apple iTune podcast help
Advice on buying MP3 players

This post edited to update links July 2, 2011

26 lessons about RBM from the 1990's remain valid today

2010-12-31T17:23:00.011-05:00

Greg Armstrong --

Lessons learned about RBM in the last century remain valid in 2011.

Implementing Results-Based Management: Lessons from the Literature – Office of the Auditor-General of Canada

Level of difficulty: Moderate
Primarily useful for: Senior Managers of partner Ministries and aid agencies
Length: Roughly 18 pages (about 9,000 words)
Most useful section: Comments on the need for a performance management culture
Limitations: Few details on implementation mechanisms

The Office of the Auditor-General of Canada deals with the practical implications of results-based management, or with the failure of agencies to use RBM appropriately, as it conducts performance audits of a large number of Canadian government agencies. The Auditor-General's website, particularly the Audit Methodology section holds several documents in the “Discussion papers” and “Studies and Tools” that are a reminder that many of the lessons learned fifteen years ago about RBM remain relevant today.

Who this is for:

26 Lessons on RBM - Reviewed by Greg Armstrong

The paper Implementing Results-Based Management: Lessons from the Literature provides a concise and relatively jargon-free summary of lessons from practical experience about how to implement results-based management. Its purpose, as the introduction notes, was to “assess what has worked and what has not worked with respect to efforts at implementing results-based management”. It is shorter and more easily read than some of the useful but much longer publications on RBM produced since, and could be useful to agency leaders wanting a reminder of where the major pitfalls lie as they attempt to implement results-based management and results-based monitoring and evaluation systems.

The lessons reported on here about implementation of results based management remain as valid today as they were in 1996 when a first draft was produced, and in 2000, when this document was released.

Lessons learned about RBM in the last century remain relevant in 2010

Many of the lessons described briefly here are derived from study of field activities of agencies from North America, Europe, and the Pacific, going back at least twenty years. The 2000 paper is based on reviews of 37 studies on lessons learned about RBM which were themselves published between 1996-1999, and builds on the earlier study, referenced briefly here, which reviewed 24 more studies produced between 1990-1995.

More recent reviews of how RBM or Management for Development Results are -- or should be --implemented in agencies such as the United Nations, such as Jody Kusek and Ray Rist’s 2004 Ten Steps to a Results-Based Monitoring and Evaluation System and Alexander McKenzie’s 2008 study on problems in implementing RBM at the UN country level build on, and elaborate many of the points made in these earlier studies, moving from generalities to more specific suggestions of how to make operational changes.

The 2000 paper from Canada's Office of the Auditor-General lists 26 lessons on how to make RBM work, and many of them repeat, and elaborate on the lessons learned earlier. The lessons on effective results-based management as they are presented here are organized around three themes:

Promoting favourable conditions for implementation of results-based management
Developing a results-based performance measurement system
Using performance information

A brief paraphrased summary of these lessons will make it obvious where there are similarities to the more detailed work on RBM and results-based monitoring and evaluation done in subsequent years. My comments are in italics:

Promoting Favourable Implementation Conditions for RBM

1. Customization of the RBM system: Simply replicating a standardised RBM system won’t work. Each organization needs a system customized to its own situation.

The literature on implementation of innovations, going back to the 1960’s confirms the need for adaptation to local situations as a key element of sustained implementation.

2. Time required to implement RBM: Rushing implementation of results-based management doesn’t work. The approach needs to be accepted within the organization, indicators take time to develop, data collection on the indicators takes more time, and results often take more time to appear than aid agencies allocate in a project cycle.

Many of the current criticisms of results-based management in aid agencies focus on the difference between the time it takes to achieve results, and aid agencies’ shorter reporting timelines.

3. Integrating RBM with existing planning: Performance measures, and indicators, should be integrated with strategic planning, tied to organizational goals, and management needs, and performance measurement and monitoring need high-level endorsement from policy makers.

Recent analyses of problems in the UN reporting systems repeat what was said in articles published as long ago as 1993. These lessons have evidently not been internalised in some agencies.

4. Indicator data collection: We should build management systems that support indicator data collection and results reporting and, where possible, build on existing data collection procedures.

5. Costs of implementing RBM: Building a useful results-based management system is not free. The costs need to be recognised and concrete budget support provided from the beginning of the process.

This is something most aid agencies have still not dealt with. They may put in place substantial internal structures to support results reporting, but shy away from providing implementing agencies with the necessary resources of time and money for things such as baseline data collection.

6. Location for RBM implementation: There are mixed messages on where to locate responsibility for coordinating implementation of RBM. Some studies suggested that putting control of the performance measurement process in the financial management or budget office, “may lead to measures that will serve the budgeting process well but will not necessary be useful for internal management". Others said that responsibility for implementation of the RBM system should be located at the programme level to bring buy-in from line managers, and yet another study made the point that the performance management system needs support from a central technical agency and leadership from senior managers.

The consensus today is that -- obviously in a perfect world -- we need all three: committed high level leadership, technical support and buy-in from line managers.

7. Pilot testing a new RBM system: Testing a new performance management system in a pilot project can be useful before large-scale implementation – if the pilot reflects the real-world system and participants.

8. Results culture: Successful implementation requires not simply new administrative systems and procedures but the development of a management culture, values and behaviour that really reflect a commitment to planning for and reporting on results.

15 years after this point was made in some analyses of implementation of results-based management, the lack of a results culture in many UN agencies was highlighted in the 2008 review of UN agency RBM at the country level, and the 2009 UNDP handbook on planning, monitoring and evaluating for development results, reiterates the old lesson that building this culture is still important for implementation of results-based management.

9. Accountability for results: Accountability for results needs to be redefined, holding implementers responsible not just for delivering outputs, but at least for contributing to results, and for reporting on what progress has been made on results, not just on delivery of outputs.

The need to focus on more than just deliverable outputs to make results-based management a reality, was mentioned in some articles in the early 1990’s, reiterated in OECD documents ten years later, yet remains an resolved issue for some aid agencies which require still, just reports on deliverables, rather than on results.

10. Who will lead implementation of RBM: Strong leadership is needed from senior managers to sustain implementation of a new performance management system.

This remains a central concern in the implementation of results based management and performance assessment. Strong and consistent leadership, committed to, and involved in the implementation of a new RBM system, remains in recent reviews of aid agency performance, such as the evaluation of RBM at UNDP, a continuing issue.

11. Stakeholder participation: Stakeholder participation in the implementation of RBM -- both from within and from outside of the organization – will strengthen sustainability, by building commitment, and pointing out possible problems before they occur.

There is now a general acceptance – in theory – of the need for stakeholder participation in the development of a results-based performance management system but, in practice, many agencies are unwilling to put the resources – again, time and money – into genuine involvement of stakeholders in analysis of problems, collection of baseline data on the problems, specification of realistic results, and ongoing data collection, analysis and reporting.

12. Technical support for RBM: Training support is needed if results-based systems are to be effectively implemented, because many people don’t have experience in results-based management. Training can also help change the organizational culture, but training also takes time. Introducing new RBM concepts can be done through short-term training and material development, but operational support for defining objectives, constructing performance indicators, using results data for reporting, and evaluation, takes time, and sustained support.

A fundamental lesson from studies dating back to the 1970’s on the implementation of complex policies and innovations, is that we must provide technical support if we want a new system, policy or innovation to be sustained – We can’t just toss it out and expect everyone else to adopt it, and use it.

Some aid agencies have moved to create internal technical support units to help their own staff cope with the adoption and implementation of results-based management, but few are willing to provide the same technical support to their stakeholders and implementation partners.

13. Evaluation expertise: Find the expertise to provide this support for management of the RBM process on a continuous basis during implementation. Often it can be found within the organization, particularly among evaluators.

14. Explain the purpose of performance management: Explain the purpose of implementing a performance management system clearly. Explain why it is needed, and the role of staff and external stakeholders.

Developing Performance Measurement Systems

15. Keep the RBM system simple: Overly complex systems are one of the biggest risks to successful implementation of results-based management. Keep the number of indicators to a few workable ones but test them, to make sure they really provide relevant data.

Most RBM systems are too complex for implementing organizations to easily adopt, internalize and implement. Yet, they need not be. Results themselves may be part of a complex system. But simpler language can be used to explain the context, problems and results, and jargon discarded, where it does not translate -- literally in language but also to real world needs of implementers and ultimately the people who are supposed to benefit from aid.

16. Standard RBM terms: Use a standard set of terms to make comparison of performance with other agencies easier.

The OECD DAC did come up with a set of harmonized RBM definitions in 2002, but donors continue to use the terms in different ways, and, as I have noted in earlier posts, have widely varying standards (if any) on how results reporting should take place. So simply using standardised terms is not itself sufficient to make performance comparisons easy.

17. Logic Models: Use of a Logic Chart helps participants and stakeholders understand the logic of results, and identify risks.

Logic Models (as some agencies refer to them) were being used, although somewhat informally, 20 years ago, in the analysis of problems and results for aid programmes. Some agencies such as CIDA have now brought the visual Logic Model to the centre of project and programme design, with some positive results. The use of the logic model does indeed make the discussion of results much more compelling for many stakeholders, than did the use of the Logical Framework.

18. Accountability for results: Make sure performance measures and reporting criteria are aligned with decision-making authority and accountability within the organization. Indicator data should not be so broad that they are useless to managers. If managers are accountable for results, then they need the power and flexibility to influence results. Managers and staff must understand what they are responsible for, and how they can influence results. If the performance management system is not seen as fair, this will undermine implementation and sustainability of results based management.

19. Credible indicator data: Data collected on indicators must be credible -- reliable and valid. Independent monitoring of data quality is needed for this.

This remains a major problem for many development projects, where donors often do not carefully examine or verify the reported indicator data.

20. Set targets: Use benchmarks and targets based on best practice to assess performance.

Agencies such as DFID and CIDA are now making more use of targets in their performance assessment frameworks.

21. Baseline data: Baseline data are needed to make the results reporting credible, and useful.

Agencies such as DFID are now concentrating on this. But many other aid agencies continue to let baseline data collection slide until late in the project or programme cycle when it is often difficult or impossible to collect. Some even focus on the reconstruction of baseline data during evaluations – a sometimes weak and often ultimately last-ditch attempt to salvage credibility from inconsistent, and unstructured results reporting.

Ultimately, of course, it is the aid agencies themselves which should collect the baseline data as they identify development problems. What data do international aid agencies have to support the assumptions that first, there is a problem, and second that a problem is likely to be something that could usefully be addressed with external assistance? All of this logically should go into project design. But once again, most aid agencies will not put the resources of time and money into project or programme design, to do what will work.

Using Performance Information

22. Making use of results data: To be credible to staff and stakeholders, performance information needs to be used – and be seen to be used. Performance information should be useful to managers and demonstrate its value.

The issue of whether decisions are based on evidence or on political or personal preferences remains important today, not just for public agencies but, as it has been recently argued, for private aid.

23. Evaluations in the RBM context: Evaluations are needed to support the implementation of results based management. “Performance information alone does not provide the complete performance picture”. Evaluations provide explanations of why results are achieved, or why problems occur. Impact evaluations can help attribute results to programmes. Where performance measurement is seen to be too costly or difficult, more frequent evaluations will be needed, but where evaluations are too expensive, a good performance measurement system can provide management with data to support decision making.

Much of this is more or less accepted wisdom now. The debate over the utility of impact evaluations, primarily related to what are sometimes their complexity and cost, continues, however.

24. Incentives for implementing RBM: Some reward for staff – financial or non financial – helps sustain change. This is part of the perception of fairness because “accountability is a two way street”. The most successful results based management systems are not punitive, but use information to help improve programmes and projects.

25. Results reporting schedule: Reports should actually use results data and regular reporting can help staff focus on results. But “an overemphasis on frequent an detailed reporting without sufficient evidence of its value for public managers, the government, parliament and the public will not meet the information needs of decision-makers.”

26. Evaluating RBM itself: The performance management system itself needs to be evaluated at regular intervals, and adjustments made.

Limitations:

This study is a synthesis (as have been many, many studies that followed it) of secondary data, a compilation of common threads, not a critical analysis of the data and not, itself, based on primary data.

It is only available, apparently, on the web page, not as a downloadable document. If you print it or convert it to an electronic document, it runs about 18 pages.

The bottom line:

The basic lessons about implementation of RBM were learned, apparently, two decades ago, and continue to be reflected throughout the universe of international aid agency documents, such as the Paris Declaration on Aid Effectiveness, but concrete action to address these lessons has been slow to follow.

This article still provides a useful summary of the major issues that need to be addressed if coherent and practical performance management systems are to be implemented in international aid organizations, and with their counterparts and implementing organizations.

Reporting on complex results: A short comment

2010-09-30T17:08:00.114-04:00

Greg Armstrong --

Change is complex -- but this is not news. Is it reasonable to expect international development programmes and projects working in complex situations to report on results? A brief comment on recent discussions.

Are Aid Agency Requirements for Reporting on Complex Results Unreasonable?

A recent post titled "Pushing Back Against Linearity" on the Aid on the Edge of Chaos Blog described a discussion among 70 development professionals at the Institute on Development Studies "...to reflect on and develop strategies for ’pushing back’ against the increasingly dominant bureaucratisation of the development agenda."

This followed a May 2010 conference in Utrecht, exploring the issues of complexity and evaluation, particularly the issue of whether complex situations, and the results associated with projects in such situations, are susceptible to quantitative impact evaluations. That conference has been described in a series of blog postings at the Evaluation Revisited website and in two blog postings by Sarah Cummings at The Giraffe.

The more recent meeting described by the Aid on the Edge of Chaos blog, and in a very brief report by Rosalind Eyben of the IDS Participation, Power and Social Change Team, which can be found at Aid on the Edge of Chaos site, appears to have focused on "pushing back" against donors insisting on results-based reporting in complex social transformation projects. This report, given its brevity, of necessity did not explore in detail all of the arguments against managing for results in complex situations but a more detailed exposition of some of these points can be found in Rosalind Eyben's earlier IDS publication Power, Mutual Accountability and Responsibility in the Practice of International Aid: A relational Approach.

Reporting on Complex Change

I think it would be a mistake to assume that the recent interest in impact evaluation means that most donors are ignorant of the complexity of development. Certainly, impact evaluations have real problems in incorporating, and "controlling for" the unknown realities and variables of complex situations. But I know very few development professionals inside donor agencies who actually express the view that change is in fact a linear process. Most agree that the best projects or programmes can do is make a contribution to change, although the bureaucratic RBM terms, often unclear results terminology and the results chains and results frameworks used by these agencies often obscure this.

Change is, indeed, complex, but this is not news. The difficulties of implementing complex innovations have been studied for years - in agricultural innovation, then more broadly in assessments of implementation of public policy in the Johnson Administration's Great Society Programmes. People like Pressman and Wildavsky, and more recently Michael Fullan have been working within this complexity for years, to find a way to achieve results in complex contexts, and to report on them.

It is reasonable that anyone using funds provided by the public think and plan clearly, and explain in a similarly clear manner what results we hope, and plan, for. When assessing results, certainly, we often find that complex situations provide unexpected results, which are also often incomplete. But at the very least we have an obligation, whatever our views of change, and of complexity, to explain what we hope to achieve, later to explain what results did occur, and whether there is a reasonable argument to be made that our work contributed to this. Whether we use randomized control groups in impact evaluation, Network Analysis, contribution analysis, the Most Significant Change process, participatory impact assessment, or any of a number of other approaches, some assessment and some reasonable attempt at reporting coherently, has to be done.

The report on the Big Push Back meeting, cites unreasonable indicators ("number of farmers contacted, number of hectares irrigated") as arguments against the type of reporting aid agencies require, but these examples are unconvincing, because, of course, they are not indicators of results at all, but indicators of completed activities. [The interim results would be changes in production, and the long term results the changes in nutrition, or health to which these activities contributed, or possibly unanticipated and possibly negative results such as economic or social dislocation, that can only be reported by villagers or farmers themselves, probably using coherent participatory or qualitative research.]

Qualitative data are, indeed, often the best sources of information on change, and I say this as someone who has used this as my primary research approach over 30 years, but they should not be used casually, on the assumption that using qualitative methods provides a quick and easy escape from reporting with quantitative data. When qualitative data are used responsibly it is in a careful and comprehensive manner, and if they are used, to be credible, they should be presented as more than simply anecdotal evidence from isolated individuals. Sociologists have been putting qualitative data together in a convincing manner for decades and so too have many development project managers.

The bottom line:

This is certainly not a one-sided debate. To a limited extent, the meeting notes reflect this, and Catherine Krell's July 2010 article on the sense of disquiet she felt, after the initial conference in Utrecht about how to balance an appreciation of complexity with the need to report on results, discusses several of the issues that must be confronted if the complexity of development is to be reflected in results reporting. It is worth noting that at this date, hers is the most recent post on the Evaluation Revisited website.

The report on the "Push Back" conference notes that one participant in the meeting "commented that too many of us are ‘averse to accounting for what we do. If we were more rigorous with our own accountability, we would not be such sitting ducks when the scary technocrats come marching in’."

Whatever process is used to frame reporting questions, collect data and report on results, the process of identifying results is in everybody's interest. It will be interesting to follow this debate.

I will review the literature on impact evaluation in a future post.

Bilateral Results Frameworks --2: USAID, DIFD, CIDA and EuropeAid

2010-08-27T00:26:00.028-04:00

Greg Armstrong --

Here's the question: Will bilateral aid agencies hold the multilaterals to account for results or not? UN agencies’ results reporting is inconsistent, and the results frameworks of SIDA, AUSAID and DANIDA remain ambiguous. This post reviews results frameworks from USAID, DFID, CIDA and EuropeAid.

Level of Difficulty: Moderate to complex

Primarily useful for: Project managers, national partners

Length: 23 documents, 1,444 p.

Most useful: CIDA RBM Guide, DFID LFA Guide and EuropeAid Capacity Development toolkit

Limitations: Mounds of bureaucratic language in many of the bilateral documents make it difficult to identify and take effective guidance from potentially useful material.

Who these materials are for

Project managers, evaluators and national partners trying to understand how USAID, DIFD, CIDA and EuropeAid define results and frame their approaches to RBM.

Background: Ambiguous results chains at the UN, and some bilateral agencies.

In my previous three posts, I examined how vague UN RBM frameworks can provide the rationale for some agencies to avoid reporting on results, to focus instead simply on describing their completed activities, and how similar ambiguities in the results frameworks and definitions from AusAid, DANIDA and SIDA , would make it difficult for them to hold the UN agencies to higher standards of results reporting. The fourth and final post in this series briefly surveys how the results frameworks of four more bilateral agencies compare to those of the OECD/DAC and UNDAF. This review is only, as I noted in past posts, of those agencies where information, in English, could be obtained in reasonable time from their own or associated -- and publicly accessible -- websites. For those who want more detail, links to the relevant donor agency RBM sites can be found at the bottom of this article.

I am proceeding from the premise, again, that “cause and effect” is not a reasonable description of what is intended by a results chain, but rather that it is a notional concept of how activities can contribute to intended results.

The USAID Results Framework

Length: 4 documents, 248 p.

Most useful: 29 pages of links and references in the Guide on Planning.

Despite its commitment to improved knowledge management, noted in the OECD DAC peer review of the United States [99 p.], last conducted in 2006, USAID remains, at least at this writing in 2010, one of the most difficult of the bilateral agencies for which to find clear RBM guidelines. USAID has also recently come under criticism for its severe editing of reports from collaborating partners.

And it is not just USAID itself which is difficult to understand, but to some extent also organizations such as the Millennium Challenge Corporation, which, again at this writing in 2010, has a largely unintelligible list of indicators for what it calls results – listing the number of farmers trained, or millions of dollars in irrigation contracts funded--in other words, completed activities--rather than listing the changes (results) these activities lead to. If you can make sense, for example, of the Tanzanian results table, you are smarter than I am, or at least you have a lot of free time, or both. [Editing note, January 2012: That results table is, as of January 2012, no longer available, and a greater emphasis now appears to be put on longer-term results, and impact evaluations at the MCC website. But, that is a review for another day.]

Among the hundreds of documents on the USAID website, there are only a few which have any kind of clear results definitions and those definitions are buried deep in bureaucratic jargon.

The tasks for USAID are, of course, complicated by the integration of its work planning with that of the State Department, and the added complexities of integrating humanitarian assistance with military operations – with all of the ambiguity that creates about the nature of results.

The USAID Guide on Planning [77 p.], updated in 2010, presents its Results Framework as moving from activities to Outputs, to Intermediate Results, and finally to Assistance Objectives -- what other agencies might call Impacts or Ultimate Outcomes. So, the USAID results chain would appear to look like this:

Activities→Outputs→Intermediate Results→Assistance Objectives

The 2006 DAC peer review noted that the USAID reporting system

“…focuses mainly on “physical deliverables” (e.g. numbers of schools, numbers of clinics, etc.). With the new orientation in US foreign policy, there is an opportunity to measure development assistance performance more in outcomes than in physical deliverables.”

By 2010, some progress may have been made on this. The Automated Directives System 200 USAID Introduction to Programming Policy [ 71 p.], revised in 2010 defines an Output as:

“A tangible, immediate, and intended product or consequence of an activity within USAID control. Examples of outputs include people fed, personnel trained, better technologies developed, and new construction.” [p. 67]

This definition seems to clearly define Outputs as completed activities.

While the USAID “Guide on Planning” definition of Outputs differs from this slightly and leaves some room for possible results as “people able to exercise a specific skill, buildings built, or better technologies developed and implemented…”, it also notes that

“…it is important to understand the difference between Outputs and results….In differentiating outputs from results, it can be useful to think of results as developmentally significant changes that affect a broad segment of society, while outputs are lower-level steps that are essential in achieving these changes.” [p. 27].

A similar distinction between results and Outputs occurs in the USAID Introduction to Programming Policy where results are defined as:

“A significant, intended, and measurable change in the condition of a customer, or a change in the host country, institutions, or other entities that will affect the customer directly or indirectly. Results are typically broader than USAID-funded outputs….”[p. 70]

This suggest that simply reporting on completed activities or products would not be acceptable within the evolving USAID context. Given that the United States provides such a large percentage of the assistance available to agencies such as UNDP, it remains to be seen whether this focus on results has in any way been communicated to the UN agencies.

The DFID Results Framework

Length: 5 documents, 378 p.

Most useful: Guidance on Using the Revised LFA

The DfID results chain appears, from the documents I have seen, to look this way:

Activities→Outputs→Purpose→Goal

DFID in 2005 had in its DFID “Guidance on Evaluation and Review” for staff [87.p] a definition of Outputs similar to those of the OECD / DAC and UN:

“The products, capital goods and services which result from a development intervention; may also include changes resulting from the intervention which are relevant to the achievement of outcomes.”

While not easily available on DFID’s own website (this link comes from the Monitoring and Evaluations News archives), the 2009 DFID How-to Note: “Guidance on Using the Revised LFA” [37 p.] changed the definition of Outputs to focus on deliverables:

“Outputs are the specific, direct deliverables of the project. These will provide the conditions necessary to achieve the Purpose”.

Examples provided in this DFID LFA guide lead to the conclusion that there is still room for looking at Outputs in the DFID context, as both completed activities and as results.

For example – deliverables or completed activities:

“Output 1: All health professionals in selected Central and District Hospitals trained on revised curriculum for patient-centred clinical care “

Possible deliverable, but also possibly a near-term result:

“Output 2: In 4 target districts Ministry of Health professionals delivering all aspects of Primary Health Care (PHC) services in partnership with NGOs and Village Health Committees “

Definitely a longer-term result, probably at the Outcome or Purpose level:

“Output 3: Selected Central and District Hospitals achieving year on year improvements in national assessments of patient-centred clinical care “ [p. 12]

This is an unnecessarily confusing mixing of real results and completed activities, in the use of one term -- “Outputs”.

According to the DFID revised LFA guide, all DFID projects are now supposed to collect baseline data on indicators for Outputs, Purpose and Goal before approval [p. 33] and to assign to each Output a weight that will “provide a clearer link to how output performance relates to project purpose performance”. This suggests that while not completely responsible for achievement of results at the purpose level, project managers are expected to report on progress at that level and assess the continuing likelihood that Outputs are making a contribution to broader achievement of results.

Of course, the fact that this document is not easily available from DFID leaves open the possibility that the ideas in it may not be in universal favour within DFID.

But there are other documents which suggest that reporting on results, and not just on activities, is important to DFID. The 2010 synthesis review of 970 DFID Project Completion Reports [124 p.] ---which may have laid the groundwork for recent criticisms of DFID performance by the new British government by criticising the lack of assessment of how projects contributed to goals, did note that “The main concern in the [Project Completion Report] process is to assess performance (achievement of the stated purpose),” [p. 36] This is clearly a focus on results, and not just on delivery of products, or completion of activities.

And as an indicator that the focus on results is being taken seriously at the political level, a recently leaked DFID memorandum on which programmes or projects should be cut, suggests that the government should be focusing on projects that can be defended “as outcome focused as possible, and will deliver value for money”, and that DFID “ will only judge ourselves against commitments and outcomes that we assess pass the fitness test.”

As Philip Dearden noted in a discussion about implementation of the new DFID logical framework on the Monitoring and Evaluation News website : “Its very important to remember that many DFID programmes are now spending huge amounts of money and we need to know what changes the money is actually going to bring about.”

The April 2010 DAC OECD peer review on British aid [130 p.] concluded that

“DFID has a strong results-based management framework, and this – combined with a purpose and performance-driven organisational culture and cohesion at the senior level – is important in ensuring effective delivery of the aid programme.”

And, even before the new British Government’s multilateral aid review, there were signs that DFID was concerned about results reporting in multilateral agencies. The 2009 Guide on Using the Revised LFA put it quite clearly:

“…DFID will have to work with the fact that multiple partners mean differences in terminology and approaches.

DFID has played a leading role in ensuring harmonisation of approaches, and is committed to continuing in this vein. However, it is important that in pursuing a harmonisation agenda, we do not relax our requirements for robust monitoring and evaluation tools.

Differences in language and approach should not be an excuse for gaps in information. In fact, the revised logframe format has already been used by DFID teams when negotiating with partners. DFID needs the information in the logframe in order to report to UK taxpayers that funds are being used in the best possible way and delivering measurable results.” [p. 16]

In part because of this, and with the UK review of aid which is about to begin, there can only be more pressure forthcoming on project managers under DfID funding to go beyond reporting on activities and products, and to monitor progress on results.

It will be interesting to see if this applies to multilateral agencies using DFID funds. [Update note, August 13, 2011: The DFID Multilateral Aid Review was completed in March 2011.]

The CIDA Results Framework

Length: 4 documents, 148 p.

Most useful: Results-Base Management at CIDA: A How-To Guide

Getting a clear idea of the CIDA Results-Based Management framework is relatively easy in comparison to some other agencies, as a new CIDA RBM system is being implemented, and most of the definitions are included in 3 documents available online at the CIDA website. The most important of these is the new CIDA RBM Guide. Other, more detailed documents, however, are so far, only available by request from CIDA’s Performance Management Division. This review deals only with those readily available, online, now.

In 2008, CIDA made changes to its results-based management policy, dropping the Log Frame (although not, as has been suggested, the Logical Framework Approach) and replacing it with a Logic Model (a change I will review in more detail in a subsequent post). The former CIDA results chain, in general use up until the end of 2009 looked like this:

Activities→Outputs→Outcomes→Impact,

The new CIDA results chain (just being implemented in 2010) is as follows:

Activities→ Outputs→Immediate Outcomes→Intermediate Outcomes→Ultimate Outcomes.

There have also, as a quick glance at these two will show, been recent changes to the RBM terminology CIDA uses. These are outlined in a useful, although rather densely phrased, policy document produced in 2008, explaining both new and old CIDA RBM terms and definitions [14 p.]. This short document is worth reading because it:

Summarizes OECD/DAC, Canadian Treasury Board and old CIDA definitions for results, activities, Outputs, and Outcomes;

Makes the case for the new definitions; and

Provides very simple examples of a range of possible results chains – both the old and the new CIDA formulations, the OECD/DAC model, and a multilateral project results chain.

In the past CIDA regarded Outputs as near-term results -- for example, changes in understanding arising out of training activities. But, as this document notes “given the almost universally accepted definition of “outputs” by donors, OECD DAC, and [The Canadian Treasury Board Secretariat] as products or services, it is necessary to readjust CIDA’s former term [for near-term results] to “immediate outcome.”

The CIDA change is useful because it makes a very clear distinction between completed activities and results. But it is based on the misconception that other donors in fact do have a clear definition of Outputs as completed activities, and do not permit results to be included in the term “Outputs”. This is something that in practice is clearly not the case for many UN agencies, for SIDA, AusAid, DANIDA, USAID and even for DFID.

The new CIDA Results-Based Management framework, like most other current aid agency documents now defines Outputs as “Direct products or services stemming from the activities of an organization, program, policy or initiative”.

It is worth noting, however, that “changes” are not included in the definition of "Outputs", and one explanation of why Outputs have been redefined as products and completed activities is that the new definition

“Clearly splits development results from products and services (outputs). This distinction should strengthen performance reporting by partners, given that it is now clear they will have to report on both outputs and outcomes” [p. 7 ]

The examples provided in CIDA Results-Based Management publications also make it clear that Outputs are completed activities or products, but not results. For example,

Activity: Build wells

Output: Wells built

Outcome: Increased access to water

Activity: Develop and deliver training on well maintenance

Output: Training on well maintenance developed and delivered

Outcome: Increased ability to maintain wells.

Results in the CIDA context are

“A describable or measurable change in state that is derived from a cause and effect relationship. Results are the same as outcomes and are further qualified as immediate, intermediate, or ultimate. all termed “Outcomes” [ p. 8]

CIDA Immediate Outcomes are near-term results phrased as changes – increases in understanding, skills or access.

CIDA Intermediate Outcomes are mid-term changes, “expected to logically occur” within the life of a project, if Immediate Outcomes are achieved, including things such as increased use of clean water, or improved trust in government.

CIDA Ultimate Outcomes are hoped for long-term changes, the justification for the project, but unlikely to be achieved during the life of a project. These refer to things such as improved health status, or reduced vulnerability of children in conflict areas.

These results categories are illustrated in more detailed documents on CIDA RBM including

“Results-Based Management at CIDA: A How-to Guide" (also available, in slightly different Micorsoft Word format as the CIDA RBM Toolkit [26 p.] , which ties together the Logic Model, indicator development, and risk assessment, and

The “CIDA Logic Model with Definitions” [1 p.] which outlines how Outputs relate to results in the new Logic Model format.

The 2007 OECD DAC peer review of Canada’s aid programme [107 p.] noted that the Results-Based Management and internal audit processes in CIDA were:

“...cumbersome, with limited differentiation in the indicators required and the processes involved for large and small programmes. While this helps to compare results among different activities, efficiency is compromised. The system might also be used to justify risk aversion rather than risk management, especially in those areas where it is more challenging to articulate measurable results (e.g. in governance)” [p. 49]

The OECD/DAC review also noted the relatively strict CIDA application of RBM to small agencies:

“ For example, an application for a small workshop organised by an NGO in Canada has to set out development results as if it were equivalent to a major bilateral programme in a partner country, with requirements to provide an impact evaluation. While providing discipline for NGO proposals may appear reasonable in theory, the practice can appear unnecessarily burdensome to the applicant.” [footnote 37]

The new CIDA RBM system is unlikely to relieve small agencies of the requirement to justify activities in terms of results, but with its simpler approach to visualising results in a Logic Model, it may be an attempt to address the issues raised by the peer review, about the unwieldy nature of the process, and to make the process of identifying results more intuitive.

While the new CIDA results framework is clear and logical, there are, of course, questions about whether it will actually be implemented, not just in plans, but in reports from project managers that are clearer and more useful than those produced before the new framework was put in place. CIDA projects have often had real difficulties in producing baseline data, and it will be interesting to see if the new framework stimulates more attention from CIDA’s own managers and from project directors, to this important element of Results-Based Management.

It will also be interesting to see if CIDA applies its standards on accountability and results reporting to multilateral agencies.

I will discuss how the new CIDA RBM framework, in particular the Logic Model, and Performance Measurement Framework, are being implemented, in a subsequent post.

The EuropeAid Results Frameworks

Length: 10 documents, 710 p.

Most useful: Results Oriented Monitoring Handbook; Glossary of Terms

Despite the obvious inference that the EU aid programme could be treated as a multilateral exercise, for the purpose of this discussion, I am treating EuropeAid as a bilateral agency. But, of course, EuropeAid itself has to account for the multiple results frameworks of its component members, and as the 2007 OECD DAC peer review of the European Union's aid programme [114 p.] noted,

"Because the Community functions both as a donor agency and as a multilateral recipient of Member State funds it is understandable that it does not allocate a large proportion of its funds to other multilateral institutions...." [p. 42]

But because it works with a multitude of member countries, it has to work with differing perspectives on results, and this is reflected in the many different results chains in EuropeAid and related documents, five of which are discussed here.

The EuropeAid site has a few pages which summarize a lot of data. This includes the EuropeAid glossary of terms related to results-based management in which the EuropeAid results chain takes this form:

Activities→ Outputs→Results→Impact

The EuropeAid glossary of RBM terms makes it clear that it neither defines, nor uses, the term “Outcome”. It simply refers to results.

EuropeAid Outputs are defined as “Goods and / or services produced / delivered by the intervention (e.g. rehabilitated road).”

EuropeAid Results are defined as “Initial change attributable to the intervention (e.g. reduced transport time and cost)” leading to

EuropeAid Impacts, defined as “Further long term change attributable to the intervention (e.g. development of trade)”.

The EuropeAid guide on Evaluation Methods [97 p.] provides examples for Outputs such as “teachers trained” or “”advice provided to groups of farmers” and for results, such as “girls benefitting from increased access to education” or “new breeding practices that prevent desertification”.

Both of these examples, among others, indicate a clear difference in this EuropeAid framework between Outputs as completed activities, and results.

But other documents, such as the 2009 EuropeAid Results Oriented Monitoring Handbook [118 p.] , (which is, for some reason, occasionally not available) cloud the picture on how “results” are defined. The Results Oriented Handbook notes, for example that

“Monitors have to fully understand the concepts and terminology used in ROM and to apply them in the correct and coherent manner. This is specially true for ‘efficiency’, ‘effectiveness’, “outcomes’ and ‘outputs’ as these terms might be used differently in other management and M&E systems.” [ p. 48]

The irony is that this document does use some words differently than those defined in the EuropeAid Glossary. While the Glossary refers only to Outputs, Results and Impact, distinguishing between Outputs and Results, the Handbook on Results Oriented Monitoring says of Outputs that they are:

“the goods and services produced; e.g. children vaccinated. In the EC’s Logframe structure these are referred to as ‘results’;” .[p. 29].

The EuropeAid Glossary specifically avoided including Outputs in results, but notes that other EC documents such as the Handbook, may use “the term be used in the wider sense.”

The EuropeAid Results Oriented Monitoring Handbook is, nevertheless, an interesting document, describing a very systematic and detailed framework for monitoring which goes far beyond completion of activities, and includes assessments of Outcomes, even Impacts, and the acquisition of information from stakeholders on relevance, effectiveness and sustainability of results. And within this document there is a distinction between

EuropeAid Outputs as completed activities (eg. Training sessions”),

EuropeAid Outcomes as intermediate results (“improved capacity of those who attended the training”) and

EuropeAid Purpose, or longer-term results, (“improvements in area of intervention due to the improved capacity of the target group”) [p. 65], which are the “specific, central highest ranking objective of the project” [p. 71], the highest level on which a project reports, but not necessarily the highest level on which it is monitored.

EuropeAid Impact as the overarching result to which a project may contribute, and justification for a project.

In answer to the question “How well is the Project achieving its planned results?” the Results Oriented Monitoring Handbook says ‘It is crucial to understand that effectiveness in this part is concerned with outcomes, not with outputs (tangible goods and services).” [p. 71]

There is a clear, and logical results chain here then, and this is reflected in the differentiation between Outputs and changes -- as results -- in the detailed Background Conclusion Sheet [p. 65-74].

Activities→Outputs→Outcomes→Purpose→Impact

The EuropeAid site website’s section on results monitoring , also has 6 very short synthesis reports on results, the most recent of which is for projects in 2007, but none of these documents makes it clear whether the monitoring is focusing on results as changes, or whether this is referring to completed activities or products. There are references to “impacts” but only as potential results, so it is not clear from any of these short reports what “results” are actually being achieved. Presumably the more detailed reports would clarify this.

But also in the past year, EuropeAid has put together, in its Tools and Methods series, two interesting, and potentially very useful documents suggesting that for capacity development EuropeAid is indeed focused on results. These include the March 2009 EuropeAid Guidelines on Making Technical Cooperation More Effective [138 p.], and the EuropeAid Toolkit for Capacity Development [82 p.] Both of these discuss a results chain, for technical cooperation, focused on capacity development in which the attention to results, not just to completed activities or delivery of products is clear. The Toolkit for Capacity Development presents the results chain for capacity development interventions this way [p. 68-70]:

Activities→sector capacity (capacity development outputs)→sector outputs (capacity development Outcomes)→Sector Outcomes→Sector Impacts

In this context as the Guidelines on Making Technical Cooperation More Effective say, “…logical frameworks for “[Capacity Development Technical Cooperation]” need to focus

on outputs and outcomes beyond the immediate deliverables by TC;” [p. 29]

It is clear from the discussions in these two guides that the sector capacity or capacity development outputs are real changes in the ability of host government agencies, for example, to deliver services more effectively. These then, whatever they are called, are results. These Guides themselves refer to the 2007 Guide on Support to Sector Programmes [119 p.]. That Guide uses the simpler results chain of

Activities→Outputs→Outcomes→Impacts,

Outputs in this document are clearly completed activities. [p. 89]

There are other, more complex, results chains referenced in five working papers on results indicators on transport, education, water and sanitation, health and agriculture, all produced in 2009, suggesting a 6-stage results chain:

Activities→Outputs→Outcomes→Specific Impacts→Intermediate Impacts→Global Impacts.

This is focused at the agency level, rather than the project level. These guides too, however, make it clear it is change that must be followed, not just completion of activities.

The European Community's aid volume is huge. As the 2007 OECD DAC peer review noted the "volume of Community ODA alone is larger than that of the World Bank’s International Development Association and several times that of the United Nations Development Programme" [p. 12] and much of it is administered or influence by the work of EuropeAID. EuropeAid should, therefore, have a very big influence on how multilateral aid agencies treat results reporting.

If the EuropeAid Glossary of RBM terms definitions, and the types of capacity results described in the 2009 EuropeAid Tools and Methods guides are used, there is a clear distinction between results and completed activities. It is also clear, despite labelling Outputs as results in the Results Oriented Monitoring Handbook, that the approach described there clearly focuses on, and in theory at least, takes the monitoring of results, not just activities, seriously.

The bottom line: Holding UN agencies to account

Despite anecdotal evidence that they are themselves trying to report on results, given the ambiguity of their own definitions of results SIDA, AusAID, DANIDA would have some difficulty in holding UN agencies to higher RBM reporting standards.

On the other hand, despite some inter-agency differences in terminology, there is now in the RBM frameworks of DFID, and CIDA, in EuropeAid's specific focus on capacity development, and perhaps, if we can trust the definitions on its website, for USAID too, little of the confusion between completed activities and results that persists in the OECD DAC and UN definitions.

There is also in these bilateral frameworks, and despite some criticisms of how they work, little space or excuse for project managers to avoid taking responsibility for reporting not just on completed activities, but on results -- explicitly as those things that have changed after the guides are produced, the training completed, or the schools or health clinics constructed.

Given the inconsistent results reporting by UN agencies described in some of my earlier posts, even those bilateral aid organizations such as DFID, CIDA, EuropeAid or USAID which do, or purport to, hold their own projects to relatively high reporting standards, could be open to criticism for providing funds to UN agencies without requiring from them similar standards on reporting.

In this context, recent moves by DFID to assess multilateral aid could focus attention more clearly on the weak UN results culture, and how bilateral agencies deal with this.

Bilateral Results Frameworks --1: SIDA, AusAid, DANIDA

2010-07-30T15:51:00.019-04:00

Greg Armstrong --

As bilateral donors consider offloading responsibility for delivering aid programmes to multilateral agencies, it pays to examine what standards these UN agencies use for results reporting. Two earlier posts looked at the relatively weak results reporting in many UN agencies, and questioned why the bilateral agencies don’t hold the UN agencies to higher standards. This third post looks at how three major bilateral donors define their results frameworks.

Level of Difficulty: Moderate to complex
Primarily useful for: Project managers, national partners
Length: 23 documents, 1,337 p.
Most useful: SIDA, AusAid and DANIDA guides to the LFA
Limitations: Mounds of bureaucratic language in many of the bilateral documents make it difficult to identify and take effective guidance from potentially useful material.

Who these materials are for

Project managers, evaluators and national partners trying to understand how SIDA, AusAid and Danida define results and frame their approaches to RBM.

Background – Confusion in UN results definitions

In my previous two posts, I examined how UN Results-Based Management frameworks can provide the rationale for some agencies to avoid reporting on results, and to focus instead simply on describing their completed activities. To a large extent this confusion arises out of the ambiguous terminology used in Results-Based-Management – words that mean different things to different agencies even as they agree to use them in common. This is complicated further in the UN case when agencies apply the same terms to near-term results at the project level as are applied at the aggregated country level. As I noted in the second post, vague definitions for results, even when they are ‘harmonised’, too frequently seem to lead unmotivated or inattentive agency heads and project managers to the path of least resistance: project reporting on completed activities and not results.

This third and a subsequent fourth post in this series briefly survey how the results frameworks of 7 bilateral aid agencies compare to those of the OECD/DAC and UNDAF. This review is only of those agencies where information, in English, could be obtained in reasonable time from their own or associated -- and publicly accessible -- websites. Those already familiar with the discussion of the problems in the UN framework may want to skip directly to the review of the bilateral agencies.

I am assuming, for the moment, a general agreement that “cause and effect” is not a reasonable description of what is intended by a results chain, but rather that it is a notional concept of how activities can contribute to intended results. In this context, this post examines how results chains are presented in documents from the OECD/DAC, SIDA, AusAid, and Danida.

USAID, DFID, CIDA and EuropeAid results frameworks will be analysed in the final post in this series.

OECD DAC RBM

The OECD DAC results chain is a common link between the bilateral and UN agencies, and it looks like this:

Activities→Outputs→Outcomes→Impact

"Outcomes", as used here, compare roughly with what the bilateral agencies work towards as development changes or results. However, it is at the Output level that ambiguity and problems occur. The approved OECD/DAC RBM terminology for Outputs, in French is “Extrants/Produit” and in Spanish it is “Producto”. In both cases, the intention is clear and stated this way, there is nothing obviously wrong with the results chain.

But the definition of (the English) Outputs provided in the OECD/DAC glossary has two elements:

First: “The products, capital goods and services which result from a development intervention;”

and then: “changes resulting from the intervention which are relevant to the achievement of outcomes.”

It is this second clause -- on “changes” -- that blurs the distinction between products or completed activities on the one hand, and results on the other.

And it is this essential lack of clarity in the original OECD/DAC definitions that is now reflected in the UN agency definitions for Outputs. Given that there are two choices here – defining Outputs as completed activities – or as real results (changes) -- and not making a clear distinction between the two, this ambiguity has provided unmotivated UN project directors the excuse to simply report on completed activities.

Bilateral Aid Agency Results Frameworks

Many of the bilateral donors reflect this same ambiguity in how they define Outputs, some specifying them as just completed activities or products, while others include the additional possibility of near-term change.

Nevertheless, whether activities and products are labelled as completed activities or as Outputs, for some bilateral aid agencies, they are not considered sufficient to constitute results reporting in results based management terms.

Thus, for example, an activity that trains 500 health workers may have Outputs (completed activities or products) of “training materials” and 500 “trained workers”. But completion of the training would not be considered a “result” by the tougher bilateral RBM standards.

Instead, most agencies would look for reports of viable results at the primary or near-term level only where it was clear what difference the training made. In this context, the result of the activity would be visible if it could be demonstrated that a substantial number of the 500 health trainees

learned something they did not know before,
changed their attitudes, or
were now working more effectively.

In the longer term, results would presumably be sought also in more systemic types of change --- in changes seen in effectiveness and relevance of the professional development system or in health services delivery policy and practice, and eventually changes in health status for the population.

The SIDA Results Framework

Length: 7 publications, 507 p.

Most useful sections: SIDA Guidebook on the LFA

The 2004 SIDA Summary of the Theory Behind the LFA, [35 p.] which I reviewed several months ago presents the SIDA results chain this way:

Activities→Results/Outputs→Project Objective/Purpose→Development objectives

This LFA guide describes Outputs as “actual, tangible results that are a direct consequence of the project’s activities”. This, on the surface, seems similar to the vague UN definitions of Outputs sometimes being products and at other times low-level results.

While the SIDA guide goes further in saying: “The outputs/results are a description of the value of the services/products produced by the project within the framework of what the project stakeholders can guarantee”, the examples the Guide provides are ambiguous. Some cite completed activities as Outputs e.g.:

Activity: “Repair old water points”; Output: “50% of existing water points…repaired”

On the other hand, other examples cite actual results as Outputs:

Activity: “Train in hygiene”; Output: “Hygienic habits of the target group improved”

This may be why a 2008 SIDA evaluation of Policy Guidance and Results-Based Management for the SIDA education sector [183 p.] noted in its comments on two programmes that:

“Even when civil society or university capacity building outcomes/ outputs are defined in log frames, performance indicators are frequently unclear and the results chain and causal relationship from activity, output and outcome are unclear.” [p.41]

A 2008 evaluation of SIDA Human Rights and Democratic Governance project results [44 p.] noted the ambiguity in how results were reported among projects – some reporting completed activities as results, under the “Output” label, and others reporting on Objectives.

“Training is by far the most frequently mentioned output; it is an output of nearly half of all the projects covered by the sample. Apart from this, the list of outputs includes, but is not limited to: policies, guidelines, studies, publications, information, seminars, study tours, infrastructure, theatre productions, and funding. Some may find that outputs, such as those found in the evaluations and studies on SIDA’s support to HR&DG, are not results but an expression of the different activities initiated with support from Sida.

That is true, but they are, nevertheless, the most tangible and direct manifestations of the interventions. And, as mentioned above, listing these as “results” is in line with SIDA’s and DAC’s usage of the concept.” [p. 11]

This report also noted that

“Rather surprisingly, the study team found that only four of the reports made a clear distinction between outputs and outcomes. In other words, the general picture of the 31 reports in the sample is that they apply this terminology rather imprecisely and give limited room for drawing any conclusions whether projects and programmes differentiate between different result levels.”

This is really not dissimilar to the general thrust of a review of “The Management of Results Information at SIDA” [18 p.] produced in 2001. Since information on

“programme expenditure, activities and outputs are covered in reports on programme progress, SIDA should make sure that counterparts’ annual reports focus on information about programme outcomes and impact.” [p. 15].

The suggested format for this report allocated roughly 80% of reporting space to the results (Outcomes and Impacts) and only about 20% to Outputs.

But although there is a move to strengthen RBM in SIDA, the OECD/DAD 2009 peer review of SIDA [123 p.] noted that “at the time of the peer review visits, many staff remained unclear what results-based management really entails in practice.” [p. 16]

The peer review also noted that

“SIDA’s very detailed manual on managing individual projects and programmes focuses strongly on planning and approval processes but provides no guidance on and makes no mention of results-based management.” [p. 59].

A new guide was planned for 2009, but I could find no sign of it on the SIDA site.

SIDA is not alone in its mixed use of Outputs, of course, and there are suggestions that it may be moving toward greater clarity. A 2007 paper by the SIDA Results Project Group “Strengthening SIDA Management for Development Results” [20 p] suggests that the agency wants reports on results, not just activities. And a 2009 study comparing accountability in four aid agencies [84 p.] found that after criticisms in 2009 by the Swedish Minister of International Cooperation that SIDA was not reporting adequately on results “the first report on Outcomes” was produced in 2009.

“SIDA is now further expected to include these new results data in its annual reports….Meanwhile the “cascading effect” of this results orientation seems to have altered the functioning of SIDA’s major partners responsible for implementing the projects: a number of interviews with these partners notably refer to the growing budget they allocate to measuring projects results.” [p. 56]

One particularly telling suggestion in reference to joint projects and partner capacity for RBM was made in the 2007 Strengthening SIDA Management for Development Results: “In countries where the capacity to generate and report results information is weak, donors should support capacity building” [p. 9]. This is a good suggestion, because projects that build results-based management capacity are rare.

The AusAID Results Framework

Length: 11 documents, 412 p.
Most useful: AusGuide 3.3: The Logical Framework Approach

The AusAID site is not easy to work with, but with some searching it is possible to find three documents -- part of the AusGuide -- relevant to how the agency defines results. The most useful of these is the 2005 AusGuide 3.3: The Logical Framework Approach [37 p.] which illustrates the AusAid results chain as:

Activities→Outputs →Component Objectives/Intermediate Results →Purpose/OutcomeGoal/Impact.

The definition of Outputs here follows the standard OECD/DAC definition as: “… the tangible products (goods and services) produced by undertaking a series of tasks as part of the planned work of the activity” and the examples provided make it clear that these are simply completed activities or products:

“…irrigation systems or water supplies constructed, areas planted/developed, children immunised, buildings or other infrastructure built, policy guidelines produced, and staff effectively trained.”

Only the last, “staff effectively trained”, suggests something approaching a result by suggesting that it is not enough to train, but that there must be some measurement of whether this was done “effectively”. This opens the door to considering change: how well the staff learned something. None of the other examples provided in the guide make it clear that Outputs are more than delivered actions: a training strategy developed; new courses designed; a series of special courses delivered; books produced or distributed.

A further complication is AusAID’s differentiation between “project Outputs” and “contractible Outputs” described in the 2002 version of AusGuidelines on the LFA [45 p] and referenced briefly in the 2005 version. This leaves the impression that project managers may only be responsible for delivery of completed activities.

But another 2005 document, an updated version of its “Promoting Practical Sustainability" [43 p] notes: “Monitoring and reporting frameworks based on tools such as the logical framework approach should look beyond the contracted activity and output levels and incorporate regular assessment of the movement towards achieving sustainable outcomes “ and follows this with some specific recommendations, including that

“The basis of payment should attach payments to outputs or milestones that are largely within the control of the contractor while encouraging the contactor to focus on their contribution to ‘outcomes’."

In theory, at least, this suggests that reports (and the projects themselves) should address much more than the simple completion of activities.

And a 2006 M&E Framework Good Practice Guide [8 p.] distinguishes between Outputs as “products, capital goods and services delivered by a development intervention to direct beneficiaries” on the one hand, and Outcomes as “short-term and medium-term results of an intervention’s outputs.”

More recent publications from AusAid differ in how they address what results are to be reported. The Office Of Development Effectiveness, established to monitor results of projects, does not make reference in its most recently published (2008) Annual Review of Development Effectiveness [74 p] to the results chain found in the AusGuide. It refers instead to the importance of achieving “objectives”, and rating their quality, through the AusAid Quality Reporting System. This apparently involves ratings by “activity” managers of “activity objectives" and “higher level strategy objectives” at entry, and later by independent evaluators, but makes no reference to any standard AusAid results chain or logic model.

Although the website entry for the Quality Reporting System says it “helps to ensure reliable, valid and robust information is available”, in the only two guides to this system that I could find, one an overview of the Quality Reporting System, [6 p.] and the other Interim Guidelines for Preparing Completion Reports [10 p.] there are no references at all to indicators, and two references to “objectives” and “key results” which may or may not be the same thing. In the ODE’s review of development effectiveness similarly, there were only three references to indicators in the context of the Australian aid programme, one of which noted that in support of rather vague “objectives” for Vietnam, “No specific indicators were identified to define the focus or scope of ambition of Australian support meant to meet these objectives.”

The December 2009 AusAid Civil Society Water, Sanitation and Hygiene Fund Guidelines [25 p.] however, clearly identify Outcomes as results, substantive changes such as “increased access to improved sanitation services” or “improved hygiene behaviour”. The same guidelines define Outputs as essentially completed activities, such as “provision of technical support” or “facilitation of dialogue”, [p. 16-17] and provide direction to NGOs applying for funds to describe their approach to “monitoring and evaluation of Outcomes”. [p. 22]

All of this is frustrating for the outsider, because it is clear from the work it was doing on contribution analysis between 2005-2009, that AusAid was genuinely trying to find practical means to document progress towards real results. As one report on the use of contribution analysis in the Fiji Education Sector program [35 p.] in 2007 noted, where previously the Program’s indicators were primarily at the Output level, after using this approach, “the new indicators subsequently provide information on progress towards and contribution to outcomes,…”. [p. 33] This report similarly provides examples of results chains somewhat different from those described in the 2005 AusGuide.

Activities→Outputs→Immediate Outcomes→Intermediate Outcomes→End Outcomes.

This was evidently not a one-shot effort because terms of reference for an AusAid evaluation [12 p.] in May of 2010, made specific reference in describing research methods to “…a basic contribution analysis and counterfactual assessment ahead of the in-country visit.”

The 2009 Peer Review of Australian Development Assistance [117 p.] states that “Performance reporting processes have now been brought together into one coherent system” but it is difficult for the average person, among the vast array of available documents on AusAid and associated websites, to find the coherence.

There probably are guides on the AusAid intranet providing clearer guidance on whether managers should now be reporting on progress towards real results at the Outcome level, but based on what is easily and publicly available, what results AusAid requires of projects, or what data it requires in support of these – whether it expects reports on completed activities and products, or on genuine results -- remains unclear.

The DANIDA Results Framework

Length: 5 documents, 418 p.
Most useful: Guide to the Logical Framework Approach

It is very difficult to get a comprehensive and updated picture of how DANIDA currently understands the results chain. The Danida website has what looks like a very practical and potentially useful guide to the Logical Framework Approach [148 p.] but it was produced in 1996, and it remains to be seen if the results chain defined there as

Activities→Outputs→Immediate Objective→Development Objective

is still valid.

The Danida definition of Outputs in this document: “Outputs are the tangible, specific and direct products of activities which largely are within the control of project management” is, nevertheless, very similar to what the OECD/DAC came up with 6 years later.

The examples provided, such as “Awareness campaign about hygiene conducted”, make it clear that outputs are seen as completed activities and products, not changes. Changes (results), in the Danida results framework, are at the Objective level statements like “Population has adequate hygiene practices”.

Somewhat ambiguously, the suggestion in the document is that project managers are responsible only for Outputs, not for Objectives, while at the same time project evaluations are to assess whether the logic of the project was valid, and whether the Outputs did contribute to the Objectives.

In this sense, the “results chain” expectation is that a project or programme will eventually report on how its Outputs contributed to Objectives. The November 2009, DANIDA Analysis of Programme and Project Completion reports 2007-2008 [76 p.], lists in the annexes the format for these reports, and they include an assessment of

“…the extent to which the programme/component has achieved the general objectives as defined in the partner programme document, and discuss the contribution by Danida to achieving the objectives” .

It is the Project and Programme Managers who must do this reporting, so we would expect that this might generate some interest in collecting data not just on Outputs or completed activities but on the Objectives (results).

But in the Ministry of Foreign Affairs Guidelines for Programme Management [87 p.] updated in 2009, while there are some passing references to Immediate Objectives in the Template for Semi-Annual and Annual Reports [p. 66], the emphasis is clearly on details about Outputs – or completed activities [p. 67-70].

The Ministry of Foreign Affairs Project status report formats for NGOs ask for

“..an account of the project objectives, indicators related to objectives, preliminary outcomes and assessment of the potential of the project for realising the project outcomes(s) established.” .

Indicators in these reports are required for Outputs, but they are also, apparently, and somewhat ambiguously, to refer to Objectives and “Outcomes”.

On the whole, it is not really clear from these sources then, to what extent DANIDA does push for reports on results, rather than completed activities.

It could be that as Denmark decentralizes responsibility for aid management, and aid reporting not just to country offices, but to Danish missions to multilateral agencies, the definitions, and interpretations of what results must be reported on, have become more diffuse.

The 2007 OECD DAC peer review of Denmark’s aid programme [103 p.] suggested that “Denmark could consider further rationalising this reporting system, as it involves many different tools and may be time consuming given embassies’ staffing constraints.” [p. 14]

The bottom line: Holding UN agencies to account

These three agencies, SIDA, AusAid and DANIDA, have a clear interest in increasing the proportion of their aid going through multilateral agencies, and are making real efforts to improve delivery of programmes. Anecdotal evidence is that they take results reporting seriously in practice. But given the ambiguity of their own definitions of results, which often, in different documents from the same agencies, conflict, SIDA, AusAID and DANIDA would have some difficulty in holding UN agencies to higher results reporting standards.

Next post: DFID, CIDA, EuropeAid and USAID

For further reading, and more original documents on SIDA, AusAID, DANIDA and the OECD DAC see:

The Denmark Ministry of Foreign Affairs Aid Management Guidelines
The AusAid publications website
The SIDA publications page (difficult to navigate)
OECD DAC peer reviews of aid agencies

Results-Based Management at the United Nations -- 2: Ambiguous results definitions

2010-06-28T18:07:00.008-04:00

--Greg Armstrong --

Problems with United Nations agency reporting on results can, in part, be attributed to ambiguous definitions of Outputs.

Level of Difficulty: Moderate-complex
Primarily useful for: Anyone trying to understand the UN’s inconsistent RBM system
Coverage: 13 papers, totalling 721 p.
Most useful components: Technical Briefs on Outcomes, Outputs, Indicators and Assumptions
Limitations: Large number of potential documents laden with bureaucratic language

Who this is for

This post, and the previous review of UN agency problems in reporting on results, is intended for bilateral aid agency representatives, national government representatives, project managers, monitors or evaluators, trying to understand the inconsistent reporting of project results by UN agencies. Because the UN documents are often lengthy, and laden with bureaucratic language, it is unlikely to be of interest to people who don’t need to work with UN agency counterparts.

Background: Problems in UN RBM

This is the second of four posts assessing UN agency results chains, results definitions and problems in reporting on results, and (in the third and fourth posts) the results frameworks bilateral aid agencies use. In the previous post, I suggested that inconsistent UN agency results reporting could, in part, be attributable to a weak results culture, and sometimes weak leadership at the country level within some UN agencies. This post reviews how ambiguous results definitions also undermine UN agencies’ credibility in results reporting.

A third post in this series will review results chains for 3 bilateral aid agencies – SIDA, AusAID, and DANIDA -- define results, and the fourth and final post will review how USAID, DFID, CIDA and EuropeAid define results and results chains.

How results are defined in the UN at the country level

Language matters. I have argued elsewhere that the terminology used in Results-Based Management is dysfunctional, largely because the jargon (Outputs, Outcomes, Impact, Objectives, Purpose) can mean many different things and, that in the context of development programming, terms used for results are intended to mean something different than they mean in day to day usage. What works in almost all project contexts, however, is to focus on change as the characteristic of a result. This is a word and a concept that works in most languages, and appeals to people’s desire for common-sense terminology.

For most bilateral donors, whatever the specific terms they use, completed activities -- often referred to as Outputs -- are not sufficient for reporting purposes. While bilateral project managers are obviously required to report on completion of activities, the real emphasis in project reports is expected to be on if and how these activities are contributing to significant changes in the short to mid-term -- changes to knowledge, attitudes, policy or professional practice.

In other words, results.

Some agencies refer to these results as Outputs, Outcomes and Impacts. Others refer to them as Objectives, or Purpose, or as Immediate, Intermediate and Ultimate Outcomes. But whatever the terms, the focus is clear: “Tell us about how the project or programme is contributing to change, not just about how you spent the money”.

The problem for bilateral and national agency partners trying to hold UN partners to reasonable standards of results based management lie, I think, in the vast number of documents dealing with results in the UN context; the ambiguity of the UN definitions of results; and the confusion about how results chains for projects relate to results chains at the country programme level for different agencies.

At the UN Development Assistance Framework level, results from different agencies are essentially being aggregated, and as the January 2010 UNDAF document "Standard Operational Format and Guidance for Reporting Progress on the UNDAF" [22 p.] made clear, the UNDAF report should be “focused on reporting results at a strategic level….”. [Editorial note: That UNDAF guide, as of January 2012, no longer appears easily accessible on the UNDAF website, but it can be obtained from the ILO website.]

Unfortunately, terms that define results for aggregations of projects at the strategic level do not necessarily work at the individual project level.

So, how does the UN, in the UNDAF and UNDG guides and technical briefs, deal with results?

UN Results Chains

Results chains describe the sequence, and the nature of links between, activities, completed activities, and near-term, mid-term and long-term results. Most practitioners agree that a direct cause and effect between activities and results is unreasonable given the wide range of intervening variables that occur in real life, but that results chains describe the general sequence of how activities can contribute to change – or results.

The several UN documents reviewed here and in the previous post variously refer to results chains (sometimes in the same document) as

Activities→Outputs→Outcomes→Impacts

Activities→Outputs→Agency Outcomes→UNDAF Outcomes→National Priority

Activities→Outputs→Country Programme Outcomes→UNDAF Outcomes→National Priority

Activities→Outputs→Agency Outcomes→Country Programme Outcomes→UNDAF Outcomes→National Priorities

“Activity Results”→Outputs→Outcome→UNDAF Outcome→National Priority

UN Agency Outputs→UNDAF Output→National Outcome→National Goal

Looking at these it is no wonder that there are differences among implementing agencies in how results are explained for projects and programmes in the UN system.

UN Outcomes

Most UN agencies use as the basis for their own definitions of results, the 2003 harmonized UNDG Results-Based Management Terminology [3 p.] which grew out of the OECD/DAC Glossary of Key Terms in Evaluation and Results-Based Management [37 p.]. “Outcomes,” the harmonized terminology states “represent changes in development conditions which occur between the completion of outputs and the achievement of impact.”

Of the hundreds of document available at the UNDG website, the most frequently referenced for elaboration on RBM terms are four technical briefs produced in 2007. These include technical briefs on Outcomes, on Outputs, on Indicators and on Assumptions and Risk.

The 2007 Technical Brief on Outcomes, [7 p.] the apparent foundation for many of the other UNDG documents on Results-Based Management, however, explained Outcomes at the country level this way -- The UN country teams have two separate, but linked, Outcomes at the country level:

UNDAF Outcomes

Country Programme Outcomes -- which incorporate individual UN agency Outcomes. These are not as clearly defined in these documents as are UNDAF Outcomes, but they appear to be seen as the changes to things such as policy or legislation, needed to facilitate long-term institutional or behaviour change. Bilateral donors might see these as mid-level results or Intermediate Outcomes, achievable to some degree over the period of a project.

So, for example, two Country Programme Outcomes of adoption or passage of Human Rights legislation and then adequate budgeting for its implementation might - it was hoped - lead to longer term UNDAF Outcomes of improved human rights in the country.

As the useful checklist in this Technical Brief on Outcomes noted on page 4, a Country Programme Outcome “…is NOT a discrete product or service, but a higher level statement of institutional or behavioural change.” The same checklist adds that a Country Programme Outcome should describe “a change which one or more UN agencies is capable of achieving over a five year period.”

[Editorial note, January 2012]: This document was replaced in February 2011 with an updated Technical Brief on Outcomes, Outputs, Indicators and Risks and Assumptions [21 p.], which maintains some, but not all, of the principles of the 2007 brief]

All of this is fairly easily understandable, and as long as the assumptions underlying all of these intended results are monitored, these definitions should open the door for UN agencies to collaborate with other donors and with national governments on solid Results-Based Management, and the reporting of results. Many bilateral aid projects also have a 5-year term so it would be reasonable to see Outcomes occurring in that period.

However, this approach is not always applied, agency-by-agency, to UN results reporting at the project level. The new 2009 UNDP Handbook on RBM, [221 p.] something I will review in more detail in another post, while it has many very useful components, says, of the scope of project evaluations, that the focus should be on:

"...Generally speaking, inputs, activities and outputs (if and how project outputs were delivered within a sector or geographic area and if direct results occurred and can be attributed to the project)* "[p. 135] .

The footnote in that quote acknowledges, however, that some large projects may have Outcomes that could be evaluated. And while later the Handbook says of project reporting that it should include “An analysis of project performance over the reporting period, including outputs produced and, where possible, information on the status of the outcome “ [p. 115] it is clear that at the project level, the priority is on reporting of Outputs.

The UNDP Handbook has a very good discussion of problem identification and stakeholder involvement in the development of a results framework which, it says, “can be particularly helpful at the project level” [p. 53]. But while the UNDP Handbook reiterates the importance of attention to results at the country level, this is less obvious at the UNDP project level:

“Since national outcomes (which require the collective efforts of two or more stakeholders) are most important, planning, monitoring and evaluation processes should focus more on the partnerships, joint programmes, joint monitoring and evaluation and collaborative efforts needed to achieve these higher level results, than on UNDP or agency outputs. This is the approach that is promoted throughout this Handbook.” [p. 14]

The problem with this is that, if attention to results from the component parts of a development programme (i.e. the projects or activities) is missing, and if project results are not properly reported, then the foundation for country-level reporting will be, at best, hypothetical.

On the other hand, a revised draft ILO RBM Guide [34 p.] notes that:

“Some mistakenly think that outputs are ends in themselves, rather than the various means to ends. RBM reminds us to shift our focus away from inputs, activities and outputs—all of which are important in the execution and implementation of work—and place it instead on clearly defined outcomes....” [p. 17]

Confusion Over UN Agency Outputs

It is at the Output level that the real confusion starts and, in turn, I think this undermines attempts to get some UN agencies to think about, or to report clearly on, results at the project level.

The 2003 document UNDG Results-Based Management Terminology -- at least in terms of Outputs -- improved upon the earlier OECD/DAC Glossary of Key Terms in Evaluation and Results-Based Management, when it defined Outputs as “The products and services which result from the completion of activities within a development intervention.”

The original OECD/DAC definition of Outputs was “ The products, capital goods and services which result from a development intervention; may also include changes resulting from the intervention which are relevant to the achievement of outcomes.”

The introduction of the “completion of activities” in the UNDG definitions, and the absence of any mention of “changes” opened the possibility that the UN would have a definition of Outputs that would help it discriminate between activities and products on the one hand (completed training, study tours, texts produced) and results (increased understanding or changed attitudes). This would be compatible with the view of many of the major bilateral donors which see Outputs as completed activities or products – and, while a necessary step in achieving results, not themselves results.

Unfortunately, the 2007 UNDAF Common Country Assessment guidelines [76 p.] complicated the issue by defining Outputs as:

“The specific products, services, or changes in processes resulting from agency cooperation”

In the 2007 Technical Brief on Outputs , [11 p.] there is yet another definition -- and the line between necessary products and the results they contribute to has been further blurred:

“Outputs are deliverables. They normally relate to operational change: changes in skills or abilities, the availability of new products and services. They are the type of results over which managers have a high degree of influence. Failure to deliver outputs is, on the face of it, a failure of the programme or project.”

In the Checklist for validating Outputs, this same document says that “The output is a new product or service, new skill or ability that can be developed and/or delivered by one UN agency working with its partners”.

The definition in the 2009 UNDP Handbook provides some conflicting examples of what Outputs are. In some cases they are clearly completed activities or products:

“Study of environment-poverty linkages completed.
“Police forces and judiciary trained….”

But in other examples, there is the hint of the changes (results) that could come out of completed activities:

“Systems and procedures implemented and competencies developed…” [p. 59]

The last example combines as an Output, a completed activity and a learning result.

The core of the UN RBM problem

The Issues Note on Results Based Management in UNDAFs [11 p.] , analysed a number of results chains and came to the conclusion that they were often illogical, with Outputs more complex and difficult to achieve than supposedly subsequent, and more advanced, Outcomes. Downgrading the complexity of Outputs makes sense, in this context, at the country level, where many results are being aggregated, but when applied to the project level, it may provide an excuse for very limited reporting on results.

And this is the core of the problem.

If we must be limited to the simplistic three-level results chains, and if jargon like “Outputs” must be used (and I am not sure it must), then at the country level, saying that changes in skills or abilities – real development results – are part of the Output, may be legitimate. This may be particularly necessary if we are aggregating results from a large number of projects and trying to fit all of their multiple results into a three-stage description, where the only terms available are Outputs, Outcomes and Impact.

But why combine two different concepts – completed activities and the changes these produce -- under one label? Why not separate the completed activities from the change? The problem of confused labels leading to inadequate reporting, it seems, must be laid at the door of a harmonization process that permitted core RBM definitions to become -- and remain -- intellectually vapid.

When we allow mixing activities and results and labelling them both “Outputs” at the project level, and then tell project managers that their primary reporting responsibility is for Outputs, it provides two potentially dysfunctional things:

A disincentive for timid or uncertain UN agency leadership to take responsibility for achieving anything risky in terms of change -- real results -- focusing instead on the logistics of organizing activities and the delivering products;

An excuse for these same agencies, if they want to, to limit themselves to generating data and reporting on the “product or service” delivery part of the definition of Outputs, rather than on genuine changes –results-- or lack of them.

Neither response is valid, however, even in the context of the UN RBM guidelines. The same UN Technical Brief on Outputs in fact, clearly argues against such timidity -- and this is important for anyone who wants to hold UN agencies to higher standards of project reporting:

“You may be tempted to list things like workshops and seminars as outputs. After all, they are deliverable and some workshops can be strategic if they gather decision takers in one room to build consensus. But, in most cases, workshops and seminars are activities rather than outputs. And remember that outputs are not completed activities – they are the tangible changes in products and services, new skills or abilities that result from the completion of several activities.”

So, the intention was honest, and indeed some UN agencies do make a genuine attempt to report on real and tangible changes or results. ILO, for example, has in its 2005 self-directed learning module on results-based management [131 p], a framework that views Outputs in part as a result. However, with what appears to be a new draft ILO RBM guide developed after 2007, the organization had changed the emphasis. While the definition of Outputs in this new ILO RBM guide is standard, combining activities and some degree of initial change, the document distinguishes more clearly between completed activities and results, and more particularly between Outputs and results.

“RBM is also significant in that it represents a shift away from the narrow focus on inputs, activities and outputs, which are factors internal to organizations. RBM moves the focus outwards towards results, which are the external changes that organizations are working to achieve and the fundamental reason why organizations exist to begin with.” [p. 4]

ILO also has its own version of the country programme results – in its Decent Country Work Programmes – where it insists that reporting at the project level focus on results and not just on completed activities.

“At the ILO, outcomes are developed: at the organization level and appear in the [programme and budget]; at the country level and appear in [country work programmes]; and at the project level and appear in various project reports and evaluations.” [ILO draft RBM Guide p. 7].

Any ambiguity there may have been in the former ILO approach to Outputs has been reduced substantially with the injunction to “Remember that targets are connected to outcomes and indicators, and never to outputs.” [p. 13]

Other agencies, however, persist -- particularly at the project level -- in reporting only on how many people they have trained, how many guides they have produced, how many meetings have been convened. While it might be argued that the UN’s focus is at the country level, and thus Outputs must incorporate some element of change (and not be confined to completed activities), it is precisely at the level of change that project reporting is not, in actual practice, consistently taking place.

The most ironic example of this I have seen was a multi-donor project on aid effectiveness that focused on improving RBM capacity in the national aid coordinating agency. Managed by a UN implementing agency, the project year after year -- and despite bilateral pleas for reports on results -- continued to report on how many people had been trained, how many workshops held, how many reports produced. It never reported on indicator-based evidence of performance management capacity improving, or even that there had been some change in understanding or attitudes towards performance management. Both results could have been demonstrated with little real effort. However, the impression left by its failure to respond to Steering Committee requests for data on change, was that the implementing agency simply was not motivated to report on results.

While UNDP’s policies and procedures for closing a project may say that a final report “should look at sustainability of the results, including the contribution to related outcomes (and the status of these outcomes) and capacity development", it is reasonable to question how this might be done in any reasonable way without the project actually collecting data on Outcome indicators. [Editorial note, January 2012: The original link to this document was removed from the UNDP website, but the reference can, as long as the document remains available, be found on page 125 of an untitled UNDP policy paper on procedures for national implementation of UNDP-supported projects.]

Conclusions: Responsibility for results reporting

Project managers may not, in some agencies, be responsible for achieving results (change, not just completed activities) at the Outcome level, but they should be held accountable for managing for, and reporting on, development results – at the very least attempting to influence results. But if we are to take management for development results seriously, and to report on this process, we need to monitor progress by looking at indicators of change – of results -- whatever label they are given.

There is a wide variation in how UN agencies treat Results-Based Management for projects -- with some reporting on actual results of their work with partners, while others limit themselves to the mundane details of how they delivered the activities. This variation might in part be a matter of confusion in the terminology of what they are responsible for. Clarifying this at the Output level, by clearly distinguishing between completed activities, and real results, is a necessary first step in improving the results culture in UN agencies.

Poor implementation of Results-Based Management must also be seen, however, as a matter of leadership. It is the UN agency heads who determine, through what they reward and what they ignore, whether managers will focus on generating and reporting results or settle for just reporting on completed activities. And it is the UN agency country leadership that sets the tone for whether project managers will take results reporting seriously.

Obscure results terminology has just allowed idle or unfocused UN agency leadership to take the path of least resistance, and report on completed activities rather than on results.

The bottom line:

The 2009 bilateral donor assessments of UNDP [46 p.] and UNICEF indicate, as I noted in my previous post, that Bilateral aid agencies’ perceptions of how UN agencies manage for development results are more critical than country partners’ views. But as the bilateral donors provide a huge amount of assistance through these agencies, it would be wise for the UN agencies to take the criticisms of how they manage for results into account. As DFID begins its multilateral spending review, careless results reporting practices may finally come under serious review by the bilateral aid agencies.

Next: The next two posts will look at Bilateral aid agency results chains, as reflected in SIDA, DANIDA, AusAID, USAID, DFID, CIDA and EuropeAid documents.