Results-Based Management Websites

Online Results-Based Management Training: The University of Wisconsin's Logic Model development course

noreply@blogger.com (Greg Armstrong) — Thu, 15 Dec 2011 17:17:00 +0000

The University of Wisconsin’s excellent online interactive Logic Model training is a valuable, easy to use introduction to RBM – and it is free.

Level of Difficulty: Easy to moderate

Primarily Useful as: An introduction to RBM and Logic Models

Length: 3-5 hours if used online, 216 pages in the PDF format

Limitations: Some of the links are out of date

Who this is for

Like other consultants and trainers who work on international development issues, I usually work with groups, either donor or implementing agencies, training 10-200 people on RBM. This reduces the cost (to them) of the training time, travel and other expenses. But readers of this site or the RBM Training website sometimes ask if I can provide training on results based management for individuals. This would, however, be so expensive that it would not be practical. Other options such as enrolling in university courses in Europe or Canada might work, but often require more time, and again, more money, than an individual might be prepared to invest.

But there is online, an excellent resource available to introduce RBM to anyone who has no experience with RBM – or, for that matter, to refresh even the most jaded RBM practitioner’s interest. This is the University of Wisconsin Extension Department’s Enhancing Program Performance with Logic Models. Developed in 2002, and put online in 2003, there is very little in this easy to use and interactive course, that is not still relevant today, in 2011, to those who are trying to understand how to use results-based management.

Format: Interactive Online learning and a downloadable PDF

This course has both an interactive online format, and for those who don’t want to, or cannot use the internet for the 2-5 hours completion of the course takes, a PDF version of the course. The real charm of this course, however, is that it is interactive, and it is the engaging nature of the interaction that cannot be replicated in the downloadable PDF. Presentations, in the course’s 7 sections, both audio and text, are supplemented with pop-up windows which participants can view, if they want additional examples or references, followed with exercises and then feedback on the answers.

University of Wisconsin Extension Online Logic Model Training
Copyright: 2002 Board of Regents, University of Wisconsin

It should not be surprising that the interactive elements are as engaging as I found them to be, because this site was developed by a team of content and technical experts, Ellen Taylor-Powell, Larry Jones and Ellen Henert at the Extension Department of the University of Wisconsin – and extension departments are almost always the group in any university the most skilled in tailoring learning events to learners’ needs and learning styles.

Each of the course’s 7 sections contains the following elements:

A Section overview page, which includes

Introductory audio
Section learning objectives
Printable section outline to help users track their progress
Content presentation
Activities that require us to put into practice the theory on the preceding pages.

A discussion of some technical limitations, and the reasonably good alternatives the course uses, are included at the end of this post.

Course Contents: Reinforcing important lessons about RBM

The course title is Enhancing Program Performance with Logic Models, but it deals with all of the associated elements of results-based management, not just the Logic Model. Nothing in it will be a major surprise to people who work regularly with RBM, but the what this course teaches users about results-based management is worth repeating, simply because these lessons are often neglected or ignored, by even the most experienced users of RBM.

The course has 7 content sections, each of which generally takes between 20-40 minutes to complete.

What is a Logic Model?

Section 1 discusses the difference between Outputs and Outcomes, the need to test assumptions, and identify risk. This section covers 20 screens in the course, and runs from pages 7-58 in the downloadable PDF. The difference is that in the PDF, transcripts of the audio from the online course, and a number of worksheets, are included as text.

This introduction to Logic Models includes, among much else, an interesting interactive Logic Model puzzle, requiring us to test our understanding of sequence, timing, risk and results, placing 20 different statement about a programme into one of 8 categories such as resources, activities, participants, Outputs, Outcomes, assumptions and external factors (or risk).

University of Wisconsin Logic Model course - interactive puzzle
Copyright: 2002, Board of Regents, University of Wisconsin

This is not dissimilar to what you can do in a group with paper and scissors, but nevertheless engaging if you are working alone, and trying to ensure you understand the ideas of sequence, scale, change and other factors.

More about Outcomes

Section 2 takes us in detail through the different types of legitimate results that we can see in at individual, group, agency, system, or community levels, and talks about why participation of stakeholders in defining Outcomes is important.

This section includes material about RBM that more than one of the UN development agencies could usefully review. Some of these agencies – not all, but some important UN agencies - get hung up on completing activities, (Outputs) and never seem to move on, in practice, to assessing whether these lead to any real change, or result (Outcomes).

Section two uses19 screens in the online course, and 32 pages in the PDF, with all of the associated supplementary material which is found in the pop-ups and links, in the online course.

More About Program Logic

Section 3 introduces (or reintroduces to forgetful RBM cognoscenti) the concept of clarifying the logic and assumptions implied in designing activities with the intention of contributing to a result. It covers theories of change, the complexity of real-life programme logic, results chains – what it calls Outcome Chains - and whether it is reasonable to claim causality – part of the current discussion on attribution of results to programme or project interventions.

It is important to note here, for those sometimes justifiably cynical about RBM who focus on complexity, that this course itself makes the point that multiple, often unplanned or unforeseen factors can affect results, not just the programme interventions. It also discusses the idea that a simple, linear logic model may not reveal enough of the factors involved in achieving results, and why more complex logic models may be needed to underpin the simple ones that we are often forced to submit to funding agencies.

Another interactive exercise here, helps us to think through the theory of action behind a programme intervention, and despite the fact that this section has only 9 screens in the online course, and 24 pages in the PDF, the complexity of the interactive exercise on page 7 means it can take well 30 or 40 minutes to finish this section.

University of Wisconsin online Logic Model course, interactive exercise testing theories of action
Copyright: 2002, Board of Regents, University of Wisconsin

This exercise is also an example of why, if you can, it is more productive in terms of learning to go to the online course first, before using the PDF, because with the PDF material there is essentially no chance to test your own understanding of the processes, before you see the “answer”.

What does a Logic Model Look Like?

Section 4 discusses why logic models, depending on the situation, purpose and culture involved, can take many different forms. There is no single format for Logic Models that suits all needs when the intellectual process of testing theories of change is involved – even though there may be, for any given donor, only one format they want to see. There are, as the course points out, situations where Logic Models may differ in size and complexity, depending on whether they are being used to test ideas for programme planning, evaluation, communication or for programme and project managers, the core of implementation activities.

Once again, note to complexity theorists, the course makes the point that logic models need not be simplistic, linear creations, but can be useful in helping agencies and individuals understand the complexity of the systems involved in interventions.

This section takes 8 screens in the online course, and 10 pages in the PDF.

How do I draw a Logic Model?

Section 5 is perhaps the most important of the course because it focuses on the real need to develop logic models as a group, not as an individual.

“As you work through this section, you will appreciate that the best way to construct a logic model is with others. While it may be quicker and easier to work alone, try not to. Many people believe that the real value of logic modeling is the PROCESS of creating one and the understanding and consensus that you build about a program as a result.”[Section 5, screen 1, PDF: p. 127]

I can only say I agree completely. In the Results Based Management training sessions I do with donor agencies, implementing organizations and national partners, feedback suggests that 80-90% of participants were absorbed by the Logic Model development process, sometimes surprised at how often the discussions reveal previously unknown or unacknowledged differences in perception among close colleagues, about what the original problem is, what risks and assumptions they have, and what reasonable results could look like.

5 different approaches to Logic Model development

All of the approaches focus on Logic Model development a group. The first four approaches are intended for planning new programmes. These can be found at the online course, by clicking on the links at the bottom of the screen on creating a logic model for a new programme or on page 136 of the PDF. All of these approaches begin with the result, and move back to activities and necessary resources.

Start with the Long-term result or Outcome, and move back through mid-term or intermediate results, to short-term results, then to the Outputs needed to achieve these, then back again to activities and finally to inputs.
Start with the long-term desired result, but then move immediately to activities, which is often the primary interest of participants, and then testing whether these will in fact, contribute to short of mid-term results which can logically relate to the long-term result.
Start with the Long-term result, then brainstorm all of the elements that will affect this – activities, short, mid-term results, participants, risks, then sort them out to see if participants agree on the relationships and sequence.
Juxtapose the situation or problem with a long-term desired change or Outcome then move back through mid-term and short term results necessary to get there, and finally to participants, activities and resources required. This looks similar to the first option, but in fact really does, in practice in groups force those with preconceptions about what the activities should be, to confront the problem clearly, to look for results and then decide what activities and resources are needed.
A fifth approach is also listed separately, on the next screen of the course, and this is for a situation where it is necessary to start logic model development with existing resources and existing activities.

I have seen this happen where there is a second phase to a project, or where there is just too much institutional inertia to reconceptualize how to approach a problem. It is essentially an approach focusing, as I see it, on a search for results to justify what is already being done – or as the this course suggests, where an “off the shelf” programme already exists: Ask about an existing programme why each activity exists, what possible changes it can lead to, and how this can relate to a newly identified problem.

This approach is sometimes necessary, in my experience, when working on RBM with universities, where activities such as degree course work and research are accepted as the core of university activities, and therefore the starting point for interventions. It can also be necessary with some government agencies which also sometimes see every problem through the paradigm of their own mandate and existing expertise.

Moving such institutions towards a genuine questioning of what is really likely to achieve results is sometimes quite difficult.

How Good is my Logic Model?

Section 6 reviews some of the pitfalls individuals or RBM workshop facilitators may encounter in the development of Logic Models, including:

Getting lost in the RBM terminology,
Focusing too much on the mechanical aspects of putting activities and results in boxes, without assessing the plausibility of the connections between activities and results,
Focusing on – or complaining about – linearity, rather than exploring the complexities that some Logic Model formats can reveal,
Confusing the development of a Logic Model with evaluation, (for which it can indeed be a useful tool, but to which its utility is not limited)
Perceiving the Logic Model as a panacea for programme or project design or implementation problems, rather than as a tool to help us find possible solutions to such problems,
Focusing on production of the paper product, but never using it in practice.

It also makes the point that different people and agencies can use – or require the use of – Logic Models, for different purposes, and that we need to be clear about what this purpose is when we determine in how much detail we will work on Logic Model development.

This section spans 10 screens in the online programme, and 16 pages in the PDF version.

Using Logic Models in Evaluation - Indicators and Measures

Section 7 reviews how understanding the original problem, assumptions, risk and the intended logic of the results chain can help agencies determine what evaluation questions to ask during different types of evaluations – questions about, and indicators relevant to:

The quality of inputs and completed activities (Outputs),
Who is participating, and what is the reach of the activities,
Assumptions underlying programme design, and selection of activities,
Whether results are actually achieved, and to what extent they can be reasonably attributed to the intervention, and
External factors –including what many agencies refer to as risk - on the achievement of results, or the failure to achieve results.

Examination of a Logic Model and how it is formed can provide the foundation for

Needs assessments (assessing the original problem, and what can be done about it),
Process evaluations (assessment of inputs and the quality of activities)
Outcome and Impact evaluations (whether changes occurred and to what extent programme or project activities, and also external factors, may have contributed to such change).

Section 7 is the longest of the course, with 20 online screens and 49 pages in the downloadable PDF. The indicator discussions run from screens 9-20 in section 7, and pages 178-205 in the PDF.

Glossaries and references on logic model development, RBM and evaluation

The course, formally ends with section 7, but it is worth taking a look at other resources included:

The useful glossary of terms related to results-based management, evaluation, data collection and data analysis. Plain language definitons for 61 terms are provided, and this can be accessed on every page of the online course from the upper right side of each screen, or from the PDF on pages 207-211.

The bibliography of references related to results based management and evaluation can also be reached by clicking on “resources” at the top of each screen or on pages 212-216 of the PDF. The bibliography includes 72 references, 8 of which have clickable links that still appear to be functional. Most of the 72 articles or references were written between 1994-2003, but it is worthwhile in particular visiting the Centers for Disease Control evaluation resources page which has a lot of very useful, accessible, and in some cases more current, guides on evaluation, logic models and data collection. The link in the bibliography is not current, but the page will automatically redirect to the new location, which I have provided in the previous sentence.

11 additional links on evaluation issues such as questionnaire design, surveys, focus groups, quasi-experimental design and other issues can be found throughout section 7 online, or on page199 of the PDF.

Finally, the course also provides links to 15 downloadable logic model worksheets or hints, in PDF and sometimes Microsoft Word format, also under “resources” at the top of each page. These can also be found scattered throughout the PDF document.

Limitations - Technical issues:

Because this course was developed in 2002, most people will have computers capable of making use of the interactive elements of the site – but there are three potential limitations to this, all of which, however, are dealt with in some way by the designers:

1. A reliable internet connection is needed to use the site, and in many of the places I, and my colleagues have work such connections can be, at times, unreliable. You might, for example, have a half hour of access to read things like this blog, but it would be frustrating to be interrupted in the middle of this online course.

Alternative provided: Users can download the 216 page PDF version of the course, which includes the text of the whole course.

2. Flash is used for the most compelling of the interactive features, such as drag and drop creation of logic models – and this, as I understand it, is unlikely to work for people using Apple products. Most of the people I work with used Windows-based computers, and have the flash player installed, so this might not be a major problem, but the flash has to be enabled, and some people, and some network administrators, do disable it for security purposes. The website provides a link to the free download of the flash player, from Adobe. Similarly popup screens and forms provide a wealth of additional detail in every section, and some web browsers may require users to enable these through security settings.

Alternative provided: The creators of the site provide an alternative to the use of flash, with some interactive elements, using links. It is not as compelling from a learning point of view as using the flash elements, but it does permit some interaction.

3. The audio portions of the presentation use the “.ram” audio format, and a download of either RealPlayer or one of the alternatives such as VLC player is required to listen.

Alternative provided: When I first came across this course, in 2008, there was, essentially just the online version but in 2010 the PDF version was produced and it is a useful reference. Some of the links in the document are out of date, but many more of them still work, and they are themselves quite useful.

All of these, and some other technical issues are addressed also in the course online help link and the Extension department provides an email link for those who may have further questions about the content or format.

The bottom line: This is an excellent introduction to results based management, focusing on logic model development, and its design is a credit to its authors' adult education abilities. It does not pretend to replace group workshops but it provides an intelligent, practical, and easy to use walk-through of the main issues in RBM. PDF technology has moved on since 2002 and it is possible to use both flash and javascript now in PDF files. It would be interesting to see what the digital media specialists at the University of Wisconsin could do with the PDF version of this course, to introduce some of the dynamic elements of the online course.

Podcasts 3: BBC's More or Less: Behind the Stats

noreply@blogger.com (Greg Armstrong) — Tue, 02 Aug 2011 04:51:00 +0000

Greg Armstrong --

This BBC Podcast brings indicator discussions to life. BBC radio's weekly podcast More or Less: Behind the Stats presents an entertaining array of examples of how intelligent people can differ in their interpretations of indicator data, and how to apply common sense to the claims made using statistics, or other quantitative data.

BBC Radio's More or Less Podcast

Level of Difficulty: Moderate and entertaining
Primarily useful for: Policy makers, senior managers, Project stakeholders, donors, project managers, project monitors, and politicians of any nationality
Length: 28 minutes (mp3 format)
Limitations: 22 recent programmes are available for download but many older programmes are available only for listening online.

Who this is for

For project managers, donors, project monitors and policy makers who need to maintain a watchful eye on how indicator data are used, More or Less, provides some interesting examples of why it pays to be skeptical of indicator validity and reliability, the claims and the interpretations put on indicator data.

Background: Surveying Results-Relevant Radio

This is the third in a series of posts discussing how audio podcasts can reinvigorate thinking on indicators and results.

The first post in this series discussed the mechanics of downloading and listening to podcasts. The second post surveyed the wide range of available podcasts from the BBC, ABC and National Public Radio of potential use to people working on results and indicators.

This post reviews what I think is the single most useful programme on world radio, for people who work with indicators: BBC's More or Less: Behind the Stats.

BBC Podcasts

BBC is not the only source of intelligent programming available on the internet - there are, as I noted in my June 2011 post, several excellent programmes available from ABC, National Public Radio, and other sources. But BBC radio has by far the widest range of podcasts to choose from.

Roughly 9 Thousand available BBC podcasts

The BBC podcast website the last time I looked (August 2, 2011) had 287 programmes available for download. Of these, roughly half usually fall into categories such as music, comedy, sports, religion or children's programming but 129 programmes fell into the "factual" podcast category. Each of these programmes has multiple -- some dozens, some hundreds -- of individual episodes available for download, or listening online.

The BBC Radio 4 website in early August 2011 listed more than 9,600 individual programme episodes, with almost 9,000 of them still available for listening in some format. This is in the "factual" category alone. Some are BBC news programmes, and despite recent cutbacks to the BBC foreign language programming, news is still available in many languages(see links at the end of this article). News programmes often cease to be available more quickly than other documentaries, for obvious reasons of topicality, but the last time I looked, there were roughly 90 available.

Some of the nearly 9,000 available factual podcasts on radio 4 focus on consumer affairs, arts history or travel.

But there are several which provide useful but also entertaining insights into the kind of work we do when we think about results and how to describe, measure or report on them.

More or Less: Behind the Stats

Of all the programmes I have found online, the most directly and consistently relevant to results-based management is More or Less: Behind the Stats. It has been hosted, since October 2007, by economist Tim Harford, the engagingly skeptical author of The Undercover Economist, The Logic of Life, most recently Adapt and many articles for The Financial Times and other publications. Tim Harford brings a common sense and clear language approach to determining whether claims for results, and the use of statistics to support such claims, are credible.

Respect for Data

In this, he continues the work of his irreverent predecessor, Andrew Dilnot, currently the chair of a U.K. Commission examining long-term health care for the elderly. Andrew Dilnot set the tone for More or Less with his no-nonsense approach to data. As The Guardian recently wrote about the Dilnot report, it is "Rich in evidence and pithy in prose" - and this is, after all, what we need more of, in all of our reporting.

Dilnot wrote a few years ago that what is important in judging political leaders' claims is "respect for data over wishful thinking", something that could be said equally of results claimed for development projects.

"If you'd prefer to be flattered by bogus numbers, to believe that the world changes when you play statistical games, or at least to act as if it does, wrote Dilnot "you are, let's be blunt, delusional and dangerous."

In a world where political leaders exhort aid workers to base -- and justify -- their programming decisions on indicator evidence, but themselves use evidence as the basis for policy only when it suits their political needs, this is something worth remembering.

120 Available episodes of More Or Less

More or Less has been produced since 2005 in association with the Open University and their site has some supplementary written material. Additional interesting material is available also on Tim Harford’s other websites.

The programme is updated for downloads every Friday during its broadcast seasons, and appears to have 2-3 broadcast seasons of 7-9 episodes each year.

The More or Less website had, as of August 2, 2011, 22 episodes available for portable listening in the standard and (easiest to download) format - MP3. (A very rudimentary introduction on how to download and listen to podcasts, is available in my May post.) Each current episode of More or Less is 28 minutes long, and they cover the period between September 2010 and May 2011, the most recent broadcasts. An additional 28 streaming episodes of More or Less are available for listening -- but not downloading -- using the BBC Player, covering the period between January 2009 and August 2010. A new season of More or Less begins in the first week of August 2011, and the September 2010 episodes will probably be archived before the new season ends.

There are also older More or Less episodes, going back to February 2003, with Tim Harford or Andrew Dilnot, but there is no point in trying to access those by clicking on the "previous programmes by year" link, because some work, and some don't. But if you go to the More or Less Archives you can get 73 more episodes. What is curious about these older archived programmes is that some - such as the earliest available episode of More or Less in February 2003, do have audio you can listen to, albeit in the sometimes problematic ram format, while others, including some later programmes, simply have a written description, but do not, as far as I can see, have an audio component. All six of the More or Less episodes broadcast between February and March 2003 have audio, for example, but none of the six episodes broadcast in January and February 2004 appear to have audio.

Nevertheless, all things considered, I estimate that that there are probably about 120 episodes of More or Less that you can listen to one way or the other and as the new season arrives there will be more.

A wide range of indicators

Each episode of More or Less usually deals with discussions of 5-6 indicator issues. Recent episodes, for example, have included discussions on indicators related to, among many other subjects

measuring child poverty
calculating civilian deaths in war zones
comparing international data on student achievement
measuring well-being
abuses of statistical significance claims
measuring the "fiscal multiplier"
how luck, and regression to the mean, can bias data interpretation
whether celebrity (or royal) marriages - or crime - lead to jumps in marriage rates
calculating the real costs of military interventions
how differences in data definitions can bias international comparisons
distinguishing between correlation and causation
how different methods of calculating "averages" can affect indicator data

Using More or Less as a research tool: What the data said about 2010

While many of the items on More or Less use examples from the United Kingdom, it is easy to see how lessons from the debunking of claims about problems or results, could be applied to other issues, and other countries.

A useful starting point for anyone wanting to test the More or Less range of issues, is the December 31, 2010 downloadable episode. The MP3 version of this episode can currently be downloaded from the main More or Less website, with 21 others, but it will soon be archived, and then you will need to go to the BBC player version of the programme, where currently the most recent 48 episodes are available, and listen online.

"The meat and drink of More or Less are the errors and connivances embedded in the statistics which fill each news bulletin" Tim Harford noted in the final programme of 2010. And this episode illustrates his point, as it produces the most important numbers of 2010, as seen by 7 people who work regularly with indicator interpretation.

Examining one episode in slightly more detail provides an example, I hope, of how we can use podcasts to stimulate ideas and, with a little effort, further research on indicators. As there are no clickable links in podcasts (at least not in this one), the references below are to the time (in minutes and seconds) into the podcast where you can find the reference.

Included in the 2010 Year-End summary of indicators in the news:

Indicator data on crime and social change

(00:56)
Daniel Franklin Executive Editor of The Economist, and editor of The World in 2010 and The World in 2011, discusses the difference between David Cameron's claims on crime and a "broken society" and what the statistics on crime rates, teen pregnancy, smoking and other issues say. Some of this is reflected also in the data on percentage of births to teenage mothers, suggested by National Statistician Jil Matheson (08:24-10:19)

It is easy to see how these discussions on falling crime rates and the implication for public policy could apply to other countries, such as Germany the U.S. or Canada

Indicators on defence spending

(02:30 into the episode)

Cathy Newman, former correspondent for the Financial Times, now political correspondent for Britain's Channel 4 news, and author of the Factcheck Blog contrasts former Prime Minister Gordon Brown's claims that defence spending rose in real terms in recent years, while data on inflation - adjusted spending suggested defence spending had fallen. She points out the difference between "cash spending" and "real spending", and she also notes that the Conservative - Liberal Democrat coalition government will be reducing the defence budget by about 8% in real terms over the next four years.

Misleading indicators on immigration

(04:25)

Tim Harford highlights the need to check indicator data sources, before making extravagant claims. He shows how Liberal Democratic party leader Nick Clegg's election debate claim that 80% of immigrants to the UK came from the European Union, was based on his party's misreading of an Economist article, which referred to students, not immigrants. The real figure Harford says, and a report in the Daily Telegraph appears to confirm, is about 39%. It is not known from any of these sources if the use of the faulty data was sloppy party research or careless use of the data in the debate.

Compared to what? Risk indicators

(07:30)

David Spiegelhalter, The Winton Professor of the Public Understanding of Risk at Cambridge University, talks about the need to put risk indicators in perspective. He examines risks for the military in Afghanistan, and compares these to the risk of riding a motorcycle on a major highway in the U.K.

Hans Rosling on the quality of indicator data

(10:30-18:10)

In the longest discussion in this episode, Hans Rosling, co-founder of the Gapminder Institute, who will be familiar to many people from his entertaining presentations on indicators , talks about how indicator data on issues such as child mortality and economic growth differ widely country to country in sub-Saharan Africa.

When questioned about the reliability of indicator data from under-funded African statistical offices, he explains why some indicator data such as child mortality and fertility rates are reliable, while other data on indicators such as maternal mortality and unemployment are not.

But, Hans Rosling says:

"It is not countries that have weak indicators. It is certain indicators that are weak for methodological reasons". (13:23)

There is much more in this interview with Rosling, on:

Indicators on fertility;
Rates of change to indicators such as child mortality, and economic growth in China;
Disaggregating indicator data and defining geographic focus for indicators;
His conclusions about the relationship between economic improvement, good governance and democracy;
The difference between ideals and advocacy, and facts.

Finding an indicator for the "slippery concept" of well-being
(18:37-20:24)

Statistician Michael Blastland former producer of More or Less, co-author with former host Andrew Dilnot of The Numbers Game: The Commonsense Guide to Understanding Numbers in the News, in Politics, and in Life , and author of a regularly published column on the BBC News online Magazine on making statistics relevant to non-statisticians, considers whether insomnia can be taken as an indicator for wellbeing - and concludes "...Well-being: this is a good illustration of how slippery a concept that is and the number of things that might have to go into it".

Dealing with slippery concepts is something development aid workers will be familiar with.

Cash as an indicator of bank viability

(20:26)

Robert Peston, Business Editor at the BBC talks about how the amount of cash banks have on hand as a percentage of what they borrowed, is an indicator of possible bank failures. Comparing British banks at the end of 2008, with British banks during the Great Depression, he comes up with some surprising information. It is presumably still relevant in 2011, but the interview never quite makes clear how

How Incomplete Data reporting undermines indicator utility

(22:50)

Ben Goldacre, a physician, the author of the Bad Science blog, a book of the same name, and a columnist for the Guardian uses the case of incompletely reported data on the drug reboxetine to illustrate how published studies on drug effectiveness provide unreliable indicators for safety and effectiveness because of the way data are both withheld, and then reported. While this brief comment focuses on this one drug, his blog has discussed several other examples of how inaccurately reported data on drug trials can be, to put it mildly, unreliable. These include, within the past year, reviews of a medicine for schizophrenia that may cause diabetes, a diabetes medication that may increase risks of heart attacks and several critiques on claims for homeopathic medicine.

This short session of More or Less with Ben Goldacre is of relevance for those involved in results reporting for any field. Checking the methods and context for any research providing us with indicator data, is something that is often neglected as indicators are used in reports on development projects. Failing to check out the sources for published results supporting results indicators in development projects, however, rarely has such immediately dangerous implications as does careless - or malevolent - use of indicators in the field of medical research.

Limitations:

Unbundling....other programmes do it:

As I noted in my previous survey of a number of results-relevant podcasts, several -- such as ABC's Counterpoint unbundle the programmes -- break them down into components that can be downloaded separately. ABC Radio's The Science Show makes the case for this when it says:

"The whole program cut up into separate stories - allows easy skipping from one story to the next so you can pick and choose".

What is also useful is that the Science show also makes transcripts available for many episdoes, so having listened to it, it is relatively easy to go back, check data and follow up.

Some (but not all) episodes of the National Public Radio show RadioLab also do this quite well, dividing an episode which has its own internal coherence, into 3-4 components available separately and with their own references. You can see examples of these, among many others, with the May 31, 2011 episode on talking to machines or the June 2009 episode on randomness and data patterns (stochasticity). There may be some BBC programmes that do unbundle their podcasts by themes, but I haven't found them yet.

Time limits:

While the 22 recent episodes (this will undoubtedly change in August 2011 with the new season) are available for download and portable listening, and readers can listen to, but not download, many others dating back to April 2003, some of the programmes prior to 2005 do not open easily. The easiest to download format - the MP3 versions currently available for the last 22 episodes - are usually only available in that format for a year. So if you want to be sure you can download and save some of the interesting episodes, it is worth skimming the site first, identifying potentially interesting episodes, and downloading them before they are archived or go to the BBC Player site. The September 2010 episodes will probably be archived soon.

Weak research links:

More or Less is relevant to those who work with results and indicators, but of course it provides just tantalizing summaries of the issues, not the whole picture. It is natural that many of us would want to do further research online to follow up on the issue summaries in each episode. This is the whole point of being online - to get access to multiple sources of data. We can, of course, do our own research, and many of the links I provided earlier in this post when I discussed the 2010 New Year's episode of More or Less, were links I did find myself, through some time-consuming research.

While the More or Less website has improved since January 2011, and particularly since the April 2011 season, it is still a disappointment that there is so little assistance on the site for further research, particularly for some of the programmes that predate 2011. The More or Less website does indeed now provide more links that it previously did, both directly and through the Open University link, but there remain occasional problems even in these links. In at least one case I noted in January 2010 the site misspelled the name of one of the guests (Ben Goldacre) and that error remains as I write this in July. Such things prove distracting, and complicate attempts to follow the story online with further search. While it is easy to make errors in writing (and this blog undoubtedly has some) there is a certain irony in finding such an error on the website of a podcast dealing with data integrity.

Other podcasts such as RadioLab have websites that provide much more in terms of links to further research, and in RadioLab's case, even have a reading list. None of these other programmes has the direct and frequent relevance to results and indicators that More or Less has, but they are better organized.

More or Less is, as I have said, the most useful of all of the programmes I have found so far, to those of us who work with results and indicators, and it is disappointing that the website does not make it easier to capitalize on the good work they have done. More links to background material on the More or Less website would be useful as would even the inclusion of metadata accessible to listeners who might want to go further.

The Bottom Line:

More or Less: Behind the Stats, is the most useful programme on the internet for people interested in how indicators are used in daily life, in policy analysis, politics and assessing results. It is worth listening to, for most people involved with results-based management. A new season of More or Less begins on August 5, 2011.

Podcasts and RBM - 2: Audio Podcasts from the BBC, Australian Broadcasting Corporation and National Public Radio

noreply@blogger.com (Greg Armstrong) — Thu, 30 Jun 2011 04:12:00 +0000

Greg Armstrong --

Radio programmes from the BBC, the Australian Broadcasting Corporation and National Public Radio provide stimulating insight into how other people work with and interpret indicators and results. The second of three posts on podcasts surveys the programmes available from these three broadcasters that are relevant to Results-Based Management discussions.

Level of Difficulty: Moderate-complex, but entertaining
Primarily useful for: Anyone who wants or needs fresh insights on results and indicators
Length: Usually Vary from 15 minutes to an hour. (mp3 format)
Limitations: Audio podcasts are difficult to reference, and follow up, compared to other media.

Who these programmes are for:

Some project managers dread discussions of indicator data, possibly because they rarely collect the data. But many project stakeholders, and those managers who do take the process of indicator development and data collection seriously, often seem energized by the discussions. Indicator development discussions reveal the different priorities stakeholders have, and what they think results really mean; and they challenge participants to think critically and creatively about data sources, data validity and the practicality of data collection. For these people, audio podcasts -- available free on the internet -- can provide though-provoking insights into both results and indicator development.

This is the second of three posts on audio podcasts. The first post, originally published May 28, 2011 dealt with the advantages, disadvantages, and mechanics of downloading, listening to, and using podcasts for RBM. This post surveys results-relevant podcasts available from the BBC, Radio Australia, and National Public Radio. The final post will review one programme, BBC’s More or Less, in more detail.

Thousands of podcasts to choose from

There are hundreds of possible programmes, and thousands of individual episodes of programmes available for listening, free on the internet. The BBC website alone had 262 separate available programmes, in January 2011, and by June had 288 programmes with material available for download. Of these, roughly were in categories such as music, comedy, sports, religion or children's programming. 122 programmes fell into BBC's "factual" category in January 2011. By June 2011 there were 133 factual programmes listed and most of these had dozens, sometimes hundreds of individual episodes available either for download or listening online -- some news programmes, in multiple languages, others about consumer affairs, arts, literature, economics or history.

The BBC radio 4 website itself suggests that there are over 9,000 episodes of different programmes just in this “factual” category available for listening in one of its formats, listing them alphabetically and by genre. Compared to the BBC podcast homepage, which organizes the available programmes into more recognizable categories, the 9,000 available podcasts may seem like a huge and unfathomable number to wade through -- but these radio 4 episodes are worth skimming. Some individual episodes buried there -- such the interesting 2008 Peer Review in the Dock -- do not appear to be listed on the podcast page.

But this blog is about Results-Based Management and, given that we all have limited time available for listening, the following are suggestions of some of the programmes I think are worth listening to for useful -- but also entertaining -- insights into the kind of work we do when we think about results and how to describe, measure or report on them:

Results-relevant Podcasts from the BBC

BBC's More or Less: Behind the Stats, is by far the programme with the most direct link to indicators and results based management, of any I have found. Each 24-minute episode usually includes 3-4 issues, all of which are directly relevant to how results and indicators can be interpreted. 22 individual episodes dating back to September 2010 are available for download as I write this, and 82 more going back to 2005 are available for listening online. I will review More or Less in more detail in my next post.

Thinking Allowed, a half-hour programme focusing on social science research, currently has a total of 228 episodes available -- 40 in downloadable MP3 format, dating back to September 2010. The Thinking Allowed Archives includes broadcasts going back as far as January 2007, using the BBC iPlayer.

Documentaries is by far the most prolifically accessible of all of the BBC podcasts. It had 88 24- minute episodes available, all downloadable, at the end of June -- and this is just for 2011. Another 660 downloadable programmes in MP3 format in the archive from 2007-2010. Finding these archived materials is not perfectly intuitive, but you can get access to them by going to the BBC factual/history link where, among many other programmes, the Documentaries for 2007, 2008, 2009 and 2010 are listed.

Material World, a BBC science programme had 41 half-hour episodes available for downloading the last time I looked, and 350 more in the archives for which the listener will require either RealPlayer or another media player such as VLC, capable of handling the Realplayer files.

Results-relevant Podcasts from the Australian Broadcasting Corporation (ABC Radio)

ABC radio’s Counterpoint, which is second only to BBC’s More or Less in its relevance to results discussions, delivers weekly one-hour programmes, and also "unbundles" the components – breaks the programme up into shorter segments which can be downloaded or listened to individually. Thus, you might want to listen to just that part of the Counterpoint February 14, 2011 broadcast on the “decline effect” -- or why much apparently validated published research can’t be trusted – but not those parts of the same broadcast dealing with Australian politics, limits to online publishing freedom, or the morality of long-term debt

Hiding the pigs…

The only quibble I have with Counterpoint’s unbundling

For example, the June 6, 2011 episode of Counterpoint included three components: “You’ve got to be rich to work for free”, “Hunters, the real conservationists” and “David Burchell: Anger, politics and the new media”. Looking at these, you might not, (and I did not) expect that one dealt with the fascinating issue of how Australians are trying to deal with 23 million highly intelligent feral pigs that are roaming the country. I’m not sure this has anything to do with RBM, but it’s interesting! When I pointed this out to my colleague Anne Bernard, who first led me to Counterpoint, and someone who listens to every episode in its entirety, her reply was “Armstrong, you have the attention span of a gnat! Just download the whole programme!”

But, if, like me, your attention span argues against downloading an hour of material just to find out about the pigs, you can, as I did, just download the feral pig episode

And if the pigs don't interest you, there are a number of other topics of potential relevance for results and indicator discussions in recent available podcasts of Counterpoint:

The two-part September 27, 2010 and October 4 2010 segments of Counterpoint questioning the reliability of expert opinion in general, and examining the particular problems of validity of case study data used in much popular management research.

The segment of the October 18, 2010 Counterpoint episode discussing levels of confidence in scientific understanding on environmental indicators.

The December 6, 2010 episode dealing with the relationship between expenditures on education and educational outcomes.

The segment of the April 18, 2011 broadcast on the dangers, and “unfortunate unintended consequences” of trying to turn qualitative assessments of excellence, into quantitative rankings.

The June 20, 2011 broadcast which contains two interesting indicator-related segments: one on how the quality of data collection instruments can affect data quality, and with it our conclusions about results; and another on what data tell us about the relative contributions to project results and organizational success made by senior management, mid-level project managers, and creative personnel.

Ockham’s Razor is another ABC programme which takes a slightly more academic approach to issues, with individuals making presentations on simple truths behind complex issues, rather than being interviewed. The programme has 240 13-minute episodes going back to January 2006 available for download, and a large number of transcripts for programmes as far back as 1997. These include, among much else, discussion of the difficulties of relating and working with a bizarre field of indicators for earthquake prediction, how simple language and basic math can bring policy debates into perspective, and why effective and simple solutions to policy issues are not implemented.

As an example of how many episodes are available, and on what variety of topics, a search for "evidence" in the Ockham’s Razor archives, produces a list of several hundred presentations.

A Results-Relevant Programme from National Public Radio

Radiolab – a programme from the U.S. National Public Radio produces a one hour episode every two weeks, and these can be downloaded as one entire episode, or you can choose, as with ABC’s Counterpoint, to download component parts of the episode, lasting10-30 minutes each. The format is much more story-telling than "More or Less" or "Counterpoint", and while dramatic liberties may sometimes be taken with the narratives, there is a lot of interesting material here. In total, by the end of June 2011, there were 46 one-hour episodes available. Not all of them are obviously related to results or indicators, but they are all worthy of attention. Two of the episodes I found particularly interesting were:

The June 2010 Radiolab episode “Oops” – which tells three stories about unintended and very negative results, growing out of projects with only good intentions, and
The October 2010 Radiolab episode on "Cities" particularly the component called “It’s Alive?” that describes how speed (of talking, and walking) can be used as an indicator of city culture and how physicists have used walking speed to predict city size, average income, crime rates and a number of other variables related to the culture of different cities.

The bottom line:

BBC’s "More or Less" and ABC’s "Counterpoint" provide a good starting point for anyone wanting a little entertainment with their results and indicator discussions. There are dozens of other programmes out there that I haven’t covered, and no doubt many more that I am not even aware of. Many of these may be of interest to you, and may also, as an incidental byproduct of your attention, provide new ways of looking at results and indicators.

Further listening

Referenced here:

BBC: More or Less: Behind the Stats
BBC: Thinking Allowed
BBC: Documentaries
BBC: Material World
ABC: Counterpoint
ABC: Ockham’s Razor
NPR: Radiolab
NPR: Krulwich Wonders

Other radio programmes of potential interest include:

BBC: The Reith Lecture Archives
BBC: Start the Week, with Andrew Marr
BBC: Four Thought
BBC: File on Four
Podcasts from the Guardian
Podcasts from the Scientific American

This post edited July 4, 2011 to update links.

Podcasts and RBM 1: How to use audio podcasts to reinvigorate thinking on indicators and results

noreply@blogger.com (Greg Armstrong) — Mon, 30 May 2011 20:43:00 +0000

Greg Armstrong --

Indicator discussions don’t have to be boring. A wide range of audio podcasts, easily accessible to listeners throughout the world, are available online from the BBC, ABC and NPR. This is the first of three posts surveying audio programmes available online, and relevant to results-based management.

Level of Difficulty: Moderate-complex, but entertaining
Primarily useful for: People who don’t have experience with downloading podcasts
Length: Usually Vary from 15 minutes to an hour. (mp3 format)
Limitations: Audio podcasts are difficult to reference, and follow up, compared to other media.

Who this post is for:

This is the first of three posts dealing with how, and which audio podcasts can be useful for people working on results frameworks and indicators. This introduction explains why podcasts can be useful, what their limitations are, and how to use them. This post is intended primarily for people who do not already know how to get access to, or use podcasts.

The second post in this series will survey the broad range of podcasts available primarily from the BBC, Radio Australia and one from the U.S. National Public Radio. The third post will review in more detail one particular programme on the BBC, More or Less which always has something useful to say about indicators.

Those readers who already know how to subscribe to, or download podcasts may wish to skip this post, and move on to the next two.

Why use audio podcasts in RBM? Because (gasp!) RBM can be boring

All of us who work with international development projects or with Results-Based Management, are familiar with the reams of paper, log frames, risk management frameworks and charts, generated in results and indicator discussions. These can put even the most enthusiastic proponents of results-based management into a coma.

But there is a range of very entertaining material available for listening, that can reinvigorate interest in how results and indicator data can be manipulated, misrepresented, forged, and in some inspiring instances, creatively interpreted --and not just in politics, or development assistance, but in daily life. For this, the audio podcasts available for download, or for online listening from a number of sources, are a useful and energizing source of not just learning, but entertainment, for people who work with results and indicator data on a regular basis.

Podcasts provide an escape from the drudgery of reading about RBM

I am a late adopter, someone slow to embrace new technologies, after an early, expensive and ultimately futile early adoption of the Betamax in the 1975. While my closest colleague has for many years been downloading to her MP3 player and listening to not just music but documentaries, and fiction, I only grudgingly started to do so a few months ago, when I began an exercise regime that put me in a boring environment for an hour a day. Music doesn’t provide the escape for me it does for many people, and I wanted to use the time productively. Early misguided attempts at reading while exercising produced unintended (but in hindsight predictably disastrous) results.

My colleague pointed me to the BBC website and its literally hundreds of podcasts; I continue to use it, but also moved on from there to the Australian Broadcasting Corporation’s smaller but worthwhile set of documentaries and then to the National Public Radio site. Now I find that an hour of exercise is intellectually productive and, best of all, entertaining. These podcasts provide insights on how other people deal with results and indicators in the real world, challenging my understanding of issues, and ways of thinking about them, and providing me with alternative approaches to data analysis, often things that I had skipped over, or forgotten in my daily reading. Some of these programmes are engrossing enough that I double my exercise time to complete them.

The drama of results and indicator discussions

The major attraction of using audio podcasts, for me, is this entertainment value. It is rare that a discussion, even on apparently boring topics related to results or indicators will make its way to an audio podcast on any of the major radio networks, unless there is an interesting or unusual twist to it. These programmes are often presented in a way that will stimulate the listener intellectually or emotionally, sometimes reawakening a dying interest in how to use data productively.

Listening to politicians or pharmaceutical manufacturers twist data, and then face the challenge of someone who knows enough to ask pointed and challenging questions, is much more interesting than reading the same discussion in a journal, a newspaper or online. Debates on issues such as health services, school quality, risk, crime, disastrously unintended results, and a wide range of other topics, can generate new ideas for people working with results frameworks, and struggling to recognize, generate or interpret convincing indicator data.

Some programmes such as WNYC’s Radio Lab deliberately dramatise the discussions to keep listeners involved, and that approach is effective. But most programmes benefit simply from focused questions, good editing, and the energy and passion of the people they are interviewing, to keep a listener’s interest. Many of them remind me of the best indicator discussions in a project context, when stakeholders understand the importance of indicators for defining results and activities that are important to them, and look forward to and passionately engage in the discussions about what they mean.

Podcast length

Most of the podcasts I listen to are about 30 minutes long. Some programmes such as ABC’s Counterpoint are an hour long, but listeners can download individual components of the episode, that might vary from 10 to 30 minutes in length. Some sites, such as The Scientific American /, have podcasts that last only a minute or two while others last roughly 15 minutes. ,

Difficulties in referencing or sharing podcast data

The primary disadvantage of using podcasts as a source of new ideas is that it is very difficult to footnote or bookmark the programmes. Only a few podcasts provide transcripts of their audio programmes and among those which do, such as ABC’s Ockham’s Razor , even fewer make use of the primary advantage of the internet – web links.

With paper, we can footnote references, drawing attention to individual words, sentences or ideas, and move back through an article to check consistency or the spelling of a name or an organization.

With electronic data, available on websites, we can provide links, from a blog such as this, so readers can jump to original or alternative sources, to document or challenge an idea, and readers can easily supplement ideas by using search engines.

But if you download a podcast, and you find a startling new idea you want to reference, while you are, for example, jogging, climbing stairs, lifting weights, or walking, how do you do it? I tried carrying a notebook and jotting down the ideas, but this is distracting and sometimes dangerous if you are exercising.
And it doesn’t work well in the rain.

In these cases the only useful way to actually use the podcast as a source of potential learning and a reference for other people, (at least as far as I know) is to listen to it on a computer, then go to the podcast home page, to note the web links to the individual podcast, and sometimes to note the running time of the particular quote within the podcast. Then, too, we can check the website’s home page and links for supplementary information.

So, while I often start now listening, for example, to BBC’s excellent More or Less as I exercise, I often end up listening to it again, in front of a computer, where I can pause the programme, make notes, rewind, or fast forward to relevant sections of the discussion – or jump to the web where I can seek supplementary information. I will be reviewing More or Less in greater detail in my third post in this series, and the difficulty in referencing individual stories in a programme will be illustrated more clearly in that post.

In any case, I assume everyone who reads this blog will have access to a computer – so it should be possible to go directly to some of the sites and programmes I list in my next post, and listen to them online.

The Mechanics of accessing and listening to podcasts

The most common format for podcasts is MP3. While not providing great sound for music (so I am told) this format is, certainly to my impaired hearing, good enough to deliver an audible conversation, debate or discussion. MP3 players such as the iPod all include software to play podcasts automatically. You can spend a lot of money on MP3 players if you want to, but there are perfectly serviceable models, such as the one I use, available in most countries for roughly $20 U.S. And if you decide you want to listen to these on the computer, any reasonably modern computer with a media player, such as the ubiquitous Windows Media Player, Apple’s ITunes or one of the many free alternative media players, will automatically start playing these programmes once you click on them.

There are also other formats, sometimes proprietary, used by individual websites. BBC, for example, while making podcasts for almost all of its programmes available in MP3 format when they are first put on the site, has a few that can only be listened to with its BBC iPlayer, online. This requires an updated flash player and I have had uneven success in using this where internet connections are slow. Some older archived BBC episodes from 2005 or earlier, may only be available in the ram format, which requires RealPlayer or an alternative, and these will still start automatically on most computers when you click on the file. A few of these BBC radio programmes, primarily music, are restricted in places like Canada, by the BBC licensing of its products.

Listeners can also subscribe to audio programmes through links on the site, or through aggregators such as iTunes, or Google Reader, or directly through links on the podcast webpage. Episodes can then be automatically delivered to the computer or mp3 player. Personally, I prefer to select the individual programmes, read the background, and download them myself, but many people prefer the convenience of automatic delivery.

There are, then, as far as I know, two primary ways of listening to some of this excellent material

a) With earphones, downloaded to an MP3 player or smartphone, or
b) Through your computer, by clicking the appropriate link.

Most websites can use all of the major web browsers, but there are sometimes minor differences in how you download a podcast using Google Chrome, or for example, Internet Explorer, Opera, Firefox or Safari. Most of the time, on most of the sites, the link to the specific audio programme or episode you want will give you fairly straightforward instructions, to either click to download, or click to listen at your computer, and where it doesn’t do this, left clicking will usually lead to the programme playing immediately on your computer while right clicking will often download the whole programme to your computer, for listening later either on the computer or on the MP3 player.

Time limits on available programmes

Websites vary widely in how long they will keep an individual podcast publicly available. Some do it as a matter of policy for three months, others for a month, some only for a week and some, such as NPR’s Krulwich Wonders -- basically a written blog -- appear to have only monthly episodes in audio, but many more episodes as written blog posts that can be read later.

In the case of the BBC podcasts, the length of time differs depending on the programme. Some are available for download for years, others for only a week, after which they may completely disappear, or be available only for immediate listening at your computer, but not for download.

As the BBC’s podcast website help page explains it

“But please don't forget that once you have downloaded a podcast episode, it is yours to keep forever and will not expire. Unfortunately, if you missed an episode and didn't download it within the period of availability we are not able to send you a copy.”

So, if you find something that is only potentially interesting, it is worthwhile downloading it for further review first to a computer, and then moving it, if you want, to an MP3 player, or simply keeping it for later listening. The files, because they are relatively low-fidelity for conversation, do not take up as much space on the computer as would higher fidelity music files.

What topics are available?

In my next post I will provide an overview of some of the most interesting radio programmes available, that are relevant to Results-Based Management.

The bottom line:

Podcasts can be difficult to work with as references, but they are stimulating additions to the dense written material we work with regularly, and can be a useful additional tool for people who think about and work with results and indicators.

Further reading on how to listen to podcasts:

BBC podcast help
ABC podcast help
Apple iTune podcast help
Advice on buying MP3 players

This post edited to update links July 2, 2011

26 lessons about RBM from the 1990's remain valid today

noreply@blogger.com (Greg Armstrong) — Fri, 31 Dec 2010 22:23:00 +0000

Greg Armstrong --

Lessons learned about RBM in the last century remain valid in 2011.

Implementing Results-Based Management: Lessons from the Literature – Office of the Auditor-General of Canada

Level of difficulty: Moderate
Primarily useful for: Senior Managers of partner Ministries and aid agencies
Length: Roughly 18 pages (about 9,000 words)
Most useful section: Comments on the need for a performance management culture
Limitations: Few details on implementation mechanisms

The Office of the Auditor-General of Canada deals with the practical implications of results-based management, or with the failure of agencies to use RBM appropriately, as it conducts performance audits of a large number of Canadian government agencies. The Auditor-General's website, particularly the Audit Methodology section holds several documents in the “Discussion papers” and “Studies and Tools” that are a reminder that many of the lessons learned fifteen years ago about RBM remain relevant today.

Who this is for:

26 Lessons on RBM - Reviewed by Greg Armstrong

The paper Implementing Results-Based Management: Lessons from the Literature provides a concise and relatively jargon-free summary of lessons from practical experience about how to implement results-based management. Its purpose, as the introduction notes, was to “assess what has worked and what has not worked with respect to efforts at implementing results-based management”. It is shorter and more easily read than some of the useful but much longer publications on RBM produced since, and could be useful to agency leaders wanting a reminder of where the major pitfalls lie as they attempt to implement results-based management and results-based monitoring and evaluation systems.

The lessons reported on here about implementation of results based management remain as valid today as they were in 1996 when a first draft was produced, and in 2000, when this document was released.

Lessons learned about RBM in the last century remain relevant in 2010

Many of the lessons described briefly here are derived from study of field activities of agencies from North America, Europe, and the Pacific, going back at least twenty years. The 2000 paper is based on reviews of 37 studies on lessons learned about RBM which were themselves published between 1996-1999, and builds on the earlier study, referenced briefly here, which reviewed 24 more studies produced between 1990-1995.

More recent reviews of how RBM or Management for Development Results are -- or should be --implemented in agencies such as the United Nations, such as Jody Kusek and Ray Rist’s 2004 Ten Steps to a Results-Based Monitoring and Evaluation System and Alexander McKenzie’s 2008 study on problems in implementing RBM at the UN country level build on, and elaborate many of the points made in these earlier studies, moving from generalities to more specific suggestions of how to make operational changes.

The 2000 paper from Canada's Office of the Auditor-General lists 26 lessons on how to make RBM work, and many of them repeat, and elaborate on the lessons learned earlier. The lessons on effective results-based management as they are presented here are organized around three themes:

Promoting favourable conditions for implementation of results-based management
Developing a results-based performance measurement system
Using performance information

A brief paraphrased summary of these lessons will make it obvious where there are similarities to the more detailed work on RBM and results-based monitoring and evaluation done in subsequent years. My comments are in italics:

Promoting Favourable Implementation Conditions for RBM

1. Customization of the RBM system: Simply replicating a standardised RBM system won’t work. Each organization needs a system customized to its own situation.

The literature on implementation of innovations, going back to the 1960’s confirms the need for adaptation to local situations as a key element of sustained implementation.

2. Time required to implement RBM: Rushing implementation of results-based management doesn’t work. The approach needs to be accepted within the organization, indicators take time to develop, data collection on the indicators takes more time, and results often take more time to appear than aid agencies allocate in a project cycle.

Many of the current criticisms of results-based management in aid agencies focus on the difference between the time it takes to achieve results, and aid agencies’ shorter reporting timelines.

3. Integrating RBM with existing planning: Performance measures, and indicators, should be integrated with strategic planning, tied to organizational goals, and management needs, and performance measurement and monitoring need high-level endorsement from policy makers.

Recent analyses of problems in the UN reporting systems repeat what was said in articles published as long ago as 1993. These lessons have evidently not been internalised in some agencies.

4. Indicator data collection: We should build management systems that support indicator data collection and results reporting and, where possible, build on existing data collection procedures.

5. Costs of implementing RBM: Building a useful results-based management system is not free. The costs need to be recognised and concrete budget support provided from the beginning of the process.

This is something most aid agencies have still not dealt with. They may put in place substantial internal structures to support results reporting, but shy away from providing implementing agencies with the necessary resources of time and money for things such as baseline data collection.

6. Location for RBM implementation: There are mixed messages on where to locate responsibility for coordinating implementation of RBM. Some studies suggested that putting control of the performance measurement process in the financial management or budget office, “may lead to measures that will serve the budgeting process well but will not necessary be useful for internal management". Others said that responsibility for implementation of the RBM system should be located at the programme level to bring buy-in from line managers, and yet another study made the point that the performance management system needs support from a central technical agency and leadership from senior managers.

The consensus today is that -- obviously in a perfect world -- we need all three: committed high level leadership, technical support and buy-in from line managers.

7. Pilot testing a new RBM system: Testing a new performance management system in a pilot project can be useful before large-scale implementation – if the pilot reflects the real-world system and participants.

8. Results culture: Successful implementation requires not simply new administrative systems and procedures but the development of a management culture, values and behaviour that really reflect a commitment to planning for and reporting on results.

15 years after this point was made in some analyses of implementation of results-based management, the lack of a results culture in many UN agencies was highlighted in the 2008 review of UN agency RBM at the country level, and the 2009 UNDP handbook on planning, monitoring and evaluating for development results, reiterates the old lesson that building this culture is still important for implementation of results-based management.

9. Accountability for results: Accountability for results needs to be redefined, holding implementers responsible not just for delivering outputs, but at least for contributing to results, and for reporting on what progress has been made on results, not just on delivery of outputs.

The need to focus on more than just deliverable outputs to make results-based management a reality, was mentioned in some articles in the early 1990’s, reiterated in OECD documents ten years later, yet remains an resolved issue for some aid agencies which require still, just reports on deliverables, rather than on results.

10. Who will lead implementation of RBM: Strong leadership is needed from senior managers to sustain implementation of a new performance management system.

This remains a central concern in the implementation of results based management and performance assessment. Strong and consistent leadership, committed to, and involved in the implementation of a new RBM system, remains in recent reviews of aid agency performance, such as the evaluation of RBM at UNDP, a continuing issue.

11. Stakeholder participation: Stakeholder participation in the implementation of RBM -- both from within and from outside of the organization – will strengthen sustainability, by building commitment, and pointing out possible problems before they occur.

There is now a general acceptance – in theory – of the need for stakeholder participation in the development of a results-based performance management system but, in practice, many agencies are unwilling to put the resources – again, time and money – into genuine involvement of stakeholders in analysis of problems, collection of baseline data on the problems, specification of realistic results, and ongoing data collection, analysis and reporting.

12. Technical support for RBM: Training support is needed if results-based systems are to be effectively implemented, because many people don’t have experience in results-based management. Training can also help change the organizational culture, but training also takes time. Introducing new RBM concepts can be done through short-term training and material development, but operational support for defining objectives, constructing performance indicators, using results data for reporting, and evaluation, takes time, and sustained support.

A fundamental lesson from studies dating back to the 1970’s on the implementation of complex policies and innovations, is that we must provide technical support if we want a new system, policy or innovation to be sustained – We can’t just toss it out and expect everyone else to adopt it, and use it.

Some aid agencies have moved to create internal technical support units to help their own staff cope with the adoption and implementation of results-based management, but few are willing to provide the same technical support to their stakeholders and implementation partners.

13. Evaluation expertise: Find the expertise to provide this support for management of the RBM process on a continuous basis during implementation. Often it can be found within the organization, particularly among evaluators.

14. Explain the purpose of performance management: Explain the purpose of implementing a performance management system clearly. Explain why it is needed, and the role of staff and external stakeholders.

Developing Performance Measurement Systems

15. Keep the RBM system simple: Overly complex systems are one of the biggest risks to successful implementation of results-based management. Keep the number of indicators to a few workable ones but test them, to make sure they really provide relevant data.

Most RBM systems are too complex for implementing organizations to easily adopt, internalize and implement. Yet, they need not be. Results themselves may be part of a complex system. But simpler language can be used to explain the context, problems and results, and jargon discarded, where it does not translate -- literally in language but also to real world needs of implementers and ultimately the people who are supposed to benefit from aid.

16. Standard RBM terms: Use a standard set of terms to make comparison of performance with other agencies easier.

The OECD DAC did come up with a set of harmonized RBM definitions in 2002, but donors continue to use the terms in different ways, and, as I have noted in earlier posts, have widely varying standards (if any) on how results reporting should take place. So simply using standardised terms is not itself sufficient to make performance comparisons easy.

17. Logic Models: Use of a Logic Chart helps participants and stakeholders understand the logic of results, and identify risks.

Logic Models (as some agencies refer to them) were being used, although somewhat informally, 20 years ago, in the analysis of problems and results for aid programmes. Some agencies such as CIDA have now brought the visual Logic Model to the centre of project and programme design, with some positive results. The use of the logic model does indeed make the discussion of results much more compelling for many stakeholders, than did the use of the Logical Framework.

18. Accountability for results: Make sure performance measures and reporting criteria are aligned with decision-making authority and accountability within the organization. Indicator data should not be so broad that they are useless to managers. If managers are accountable for results, then they need the power and flexibility to influence results. Managers and staff must understand what they are responsible for, and how they can influence results. If the performance management system is not seen as fair, this will undermine implementation and sustainability of results based management.

19. Credible indicator data: Data collected on indicators must be credible -- reliable and valid. Independent monitoring of data quality is needed for this.

This remains a major problem for many development projects, where donors often do not carefully examine or verify the reported indicator data.

20. Set targets: Use benchmarks and targets based on best practice to assess performance.

Agencies such as DFID and CIDA are now making more use of targets in their performance assessment frameworks.

21. Baseline data: Baseline data are needed to make the results reporting credible, and useful.

Agencies such as DFID are now concentrating on this. But many other aid agencies continue to let baseline data collection slide until late in the project or programme cycle when it is often difficult or impossible to collect. Some even focus on the reconstruction of baseline data during evaluations – a sometimes weak and often ultimately last-ditch attempt to salvage credibility from inconsistent, and unstructured results reporting.

Ultimately, of course, it is the aid agencies themselves which should collect the baseline data as they identify development problems. What data do international aid agencies have to support the assumptions that first, there is a problem, and second that a problem is likely to be something that could usefully be addressed with external assistance? All of this logically should go into project design. But once again, most aid agencies will not put the resources of time and money into project or programme design, to do what will work.

Using Performance Information

22. Making use of results data: To be credible to staff and stakeholders, performance information needs to be used – and be seen to be used. Performance information should be useful to managers and demonstrate its value.

The issue of whether decisions are based on evidence or on political or personal preferences remains important today, not just for public agencies but, as it has been recently argued, for private aid.

23. Evaluations in the RBM context: Evaluations are needed to support the implementation of results based management. “Performance information alone does not provide the complete performance picture”. Evaluations provide explanations of why results are achieved, or why problems occur. Impact evaluations can help attribute results to programmes. Where performance measurement is seen to be too costly or difficult, more frequent evaluations will be needed, but where evaluations are too expensive, a good performance measurement system can provide management with data to support decision making.

Much of this is more or less accepted wisdom now. The debate over the utility of impact evaluations, primarily related to what are sometimes their complexity and cost, continues, however.

24. Incentives for implementing RBM: Some reward for staff – financial or non financial – helps sustain change. This is part of the perception of fairness because “accountability is a two way street”. The most successful results based management systems are not punitive, but use information to help improve programmes and projects.

25. Results reporting schedule: Reports should actually use results data and regular reporting can help staff focus on results. But “an overemphasis on frequent an detailed reporting without sufficient evidence of its value for public managers, the government, parliament and the public will not meet the information needs of decision-makers.”

26. Evaluating RBM itself: The performance management system itself needs to be evaluated at regular intervals, and adjustments made.

Limitations:

This study is a synthesis (as have been many, many studies that followed it) of secondary data, a compilation of common threads, not a critical analysis of the data and not, itself, based on primary data.

It is only available, apparently, on the web page, not as a downloadable document. If you print it or convert it to an electronic document, it runs about 18 pages.

The bottom line:

The basic lessons about implementation of RBM were learned, apparently, two decades ago, and continue to be reflected throughout the universe of international aid agency documents, such as the Paris Declaration on Aid Effectiveness, but concrete action to address these lessons has been slow to follow.

This article still provides a useful summary of the major issues that need to be addressed if coherent and practical performance management systems are to be implemented in international aid organizations, and with their counterparts and implementing organizations.

Reporting on complex results: A short comment

noreply@blogger.com (Greg Armstrong) — Thu, 30 Sep 2010 21:08:00 +0000

Greg Armstrong --

Change is complex -- but this is not news. Is it reasonable to expect international development programmes and projects working in complex situations to report on results? A brief comment on recent discussions.

Are Aid Agency Requirements for Reporting on Complex Results Unreasonable?

A recent post titled "Pushing Back Against Linearity" on the Aid on the Edge of Chaos Blog described a discussion among 70 development professionals at the Institute on Development Studies "...to reflect on and develop strategies for ’pushing back’ against the increasingly dominant bureaucratisation of the development agenda."

This followed a May 2010 conference in Utrecht, exploring the issues of complexity and evaluation, particularly the issue of whether complex situations, and the results associated with projects in such situations, are susceptible to quantitative impact evaluations. That conference has been described in a series of blog postings at the Evaluation Revisited website and in two blog postings by Sarah Cummings at The Giraffe.

The more recent meeting described by the Aid on the Edge of Chaos blog, and in a very brief report by Rosalind Eyben of the IDS Participation, Power and Social Change Team, which can be found at Aid on the Edge of Chaos site, appears to have focused on "pushing back" against donors insisting on results-based reporting in complex social transformation projects. This report, given its brevity, of necessity did not explore in detail all of the arguments against managing for results in complex situations but a more detailed exposition of some of these points can be found in Rosalind Eyben's earlier IDS publication Power, Mutual Accountability and Responsibility in the Practice of International Aid: A relational Approach.

Reporting on Complex Change

I think it would be a mistake to assume that the recent interest in impact evaluation means that most donors are ignorant of the complexity of development. Certainly, impact evaluations have real problems in incorporating, and "controlling for" the unknown realities and variables of complex situations. But I know very few development professionals inside donor agencies who actually express the view that change is in fact a linear process. Most agree that the best projects or programmes can do is make a contribution to change, although the bureaucratic RBM terms, often unclear results terminology and the results chains and results frameworks used by these agencies often obscure this.

Change is, indeed, complex, but this is not news. The difficulties of implementing complex innovations have been studied for years - in agricultural innovation, then more broadly in assessments of implementation of public policy in the Johnson Administration's Great Society Programmes. People like Pressman and Wildavsky, and more recently Michael Fullan have been working within this complexity for years, to find a way to achieve results in complex contexts, and to report on them.

It is reasonable that anyone using funds provided by the public think and plan clearly, and explain in a similarly clear manner what results we hope, and plan, for. When assessing results, certainly, we often find that complex situations provide unexpected results, which are also often incomplete. But at the very least we have an obligation, whatever our views of change, and of complexity, to explain what we hope to achieve, later to explain what results did occur, and whether there is a reasonable argument to be made that our work contributed to this. Whether we use randomized control groups in impact evaluation, Network Analysis, contribution analysis, the Most Significant Change process, participatory impact assessment, or any of a number of other approaches, some assessment and some reasonable attempt at reporting coherently, has to be done.

The report on the Big Push Back meeting, cites unreasonable indicators ("number of farmers contacted, number of hectares irrigated") as arguments against the type of reporting aid agencies require, but these examples are unconvincing, because, of course, they are not indicators of results at all, but indicators of completed activities. [The interim results would be changes in production, and the long term results the changes in nutrition, or health to which these activities contributed, or possibly unanticipated and possibly negative results such as economic or social dislocation, that can only be reported by villagers or farmers themselves, probably using coherent participatory or qualitative research.]

Qualitative data are, indeed, often the best sources of information on change, and I say this as someone who has used this as my primary research approach over 30 years, but they should not be used casually, on the assumption that using qualitative methods provides a quick and easy escape from reporting with quantitative data. When qualitative data are used responsibly it is in a careful and comprehensive manner, and if they are used, to be credible, they should be presented as more than simply anecdotal evidence from isolated individuals. Sociologists have been putting qualitative data together in a convincing manner for decades and so too have many development project managers.

The bottom line:

This is certainly not a one-sided debate. To a limited extent, the meeting notes reflect this, and Catherine Krell's July 2010 article on the sense of disquiet she felt, after the initial conference in Utrecht about how to balance an appreciation of complexity with the need to report on results, discusses several of the issues that must be confronted if the complexity of development is to be reflected in results reporting. It is worth noting that at this date, hers is the most recent post on the Evaluation Revisited website.

The report on the "Push Back" conference notes that one participant in the meeting "commented that too many of us are ‘averse to accounting for what we do. If we were more rigorous with our own accountability, we would not be such sitting ducks when the scary technocrats come marching in’."

Whatever process is used to frame reporting questions, collect data and report on results, the process of identifying results is in everybody's interest. It will be interesting to follow this debate.

I will review the literature on impact evaluation in a future post.

Bilateral Results Frameworks --2: USAID, DIFD, CIDA and EuropeAid

noreply@blogger.com (Greg Armstrong) — Fri, 27 Aug 2010 04:26:00 +0000

Greg Armstrong --

Here's the question: Will bilateral aid agencies hold the multilaterals to account for results or not? UN agencies’ results reporting is inconsistent, and the results frameworks of SIDA, AUSAID and DANIDA remain ambiguous. This post reviews results frameworks from USAID, DFID, CIDA and EuropeAid.

Level of Difficulty: Moderate to complex

Primarily useful for: Project managers, national partners

Length: 23 documents, 1,444 p.

Most useful: CIDA RBM Guide, DFID LFA Guide and EuropeAid Capacity Development toolkit

Limitations: Mounds of bureaucratic language in many of the bilateral documents make it difficult to identify and take effective guidance from potentially useful material.

Who these materials are for

Project managers, evaluators and national partners trying to understand how USAID, DIFD, CIDA and EuropeAid define results and frame their approaches to RBM.

Background: Ambiguous results chains at the UN, and some bilateral agencies.

In my previous three posts, I examined how vague UN RBM frameworks can provide the rationale for some agencies to avoid reporting on results, to focus instead simply on describing their completed activities, and how similar ambiguities in the results frameworks and definitions from AusAid, DANIDA and SIDA , would make it difficult for them to hold the UN agencies to higher standards of results reporting. The fourth and final post in this series briefly surveys how the results frameworks of four more bilateral agencies compare to those of the OECD/DAC and UNDAF. This review is only, as I noted in past posts, of those agencies where information, in English, could be obtained in reasonable time from their own or associated -- and publicly accessible -- websites. For those who want more detail, links to the relevant donor agency RBM sites can be found at the bottom of this article.

I am proceeding from the premise, again, that “cause and effect” is not a reasonable description of what is intended by a results chain, but rather that it is a notional concept of how activities can contribute to intended results.

The USAID Results Framework

Length: 4 documents, 248 p.

Most useful: 29 pages of links and references in the Guide on Planning.

Despite its commitment to improved knowledge management, noted in the OECD DAC peer review of the United States [99 p.], last conducted in 2006, USAID remains one of the most difficult of the bilateral agencies for which to find clear RBM guidelines. USAID has also recently come under criticism for its severe editing of reports from collaborating partners.

And it is not just USAID itself which is difficult to understand, but also organizations such as the Millennium Challenge Corporation, which has a largely unintelligible list of indicators for what it calls results – listing the number of farmers trained, or millions of dollars in irrigation contracts funded--in other words, completed activities--rather than listing the changes (results) these activities lead to. If you can make sense, for example, of the Tanzanian results table, you are smarter than I am, or at least you have a lot of free time, or both.

Among the hundreds of documents on the USAID website, there are only a few which have any kind of clear results definitions and those definitions are buried deep in bureaucratic jargon.

The tasks for USAID are, of course, complicated by the integration of its work planning with that of the State Department, and the added complexities of integrating humanitarian assistance with military operations – with all of the ambiguity that creates about the nature of results.

The USAID Guide on Planning [77 p.], updated in 2010, presents its Results Framework as moving from activities to Outputs, to Intermediate Results, and finally to Assistance Objectives -- what other agencies might call Impacts or Ultimate Outcomes. So, the USAID results chain would appear to look like this:

Activities→Outputs→Intermediate Results→Assistance Objectives

The 2006 DAC peer review noted that the USAID reporting system

“…focuses mainly on “physical deliverables” (e.g. numbers of schools, numbers of clinics, etc.). With the new orientation in US foreign policy, there is an opportunity to measure development assistance performance more in outcomes than in physical deliverables.”

By 2010, some progress may have been made on this. The Automated Directives System 200 USAID Introduction to Programming Policy [ 71 p.], revised in 2010 defines an Output as:

“A tangible, immediate, and intended product or consequence of an activity within USAID control. Examples of outputs include people fed, personnel trained, better technologies developed, and new construction.” [p. 67]

This definition seems to clearly define Outputs as completed activities.

While the USAID “Guide on Planning” definition of Outputs differs from this slightly and leaves some room for possible results as “people able to exercise a specific skill, buildings built, or better technologies developed and implemented…”, it also notes that

“…it is important to understand the difference between Outputs and results….In differentiating outputs from results, it can be useful to think of results as developmentally significant changes that affect a broad segment of society, while outputs are lower-level steps that are essential in achieving these changes.” [p. 27].

A similar distinction between results and Outputs occurs in the USAID Introduction to Programming Policy where results are defined as:

“A significant, intended, and measurable change in the condition of a customer, or a change in the host country, institutions, or other entities that will affect the customer directly or indirectly. Results are typically broader than USAID-funded outputs….”[p. 70]

This suggest that simply reporting on completed activities or products would not be acceptable within the evolving USAID context. Given that the United States provides such a large percentage of the assistance available to agencies such as UNDP, it remains to be seen whether this focus on results has in any way been communicated to the UN agencies.

The DFID Results Framework

Length: 5 documents, 378 p.

Most useful: Guidance on Using the Revised LFA

The DfID results chain appears, from the documents I have seen, to look this way:

Activities→Outputs→Purpose→Goal

DFID in 2005 had in its DFID “Guidance on Evaluation and Review” for staff [87.p] a definition of Outputs similar to those of the OECD / DAC and UN:

“The products, capital goods and services which result from a development intervention; may also include changes resulting from the intervention which are relevant to the achievement of outcomes.”

While not easily available on DFID’s own website (this link comes from the Monitoring and Evaluations News archives), the 2009 DFID How-to Note: “Guidance on Using the Revised LFA” [37 p.] changed the definition of Outputs to focus on deliverables:

“Outputs are the specific, direct deliverables of the project. These will provide the conditions necessary to achieve the Purpose”.

Examples provided in this DFID LFA guide lead to the conclusion that there is still room for looking at Outputs in the DFID context, as both completed activities and as results.

For example – deliverables or completed activities:

“Output 1: All health professionals in selected Central and District Hospitals trained on revised curriculum for patient-centred clinical care “

Possible deliverable, but also possibly a near-term result:

“Output 2: In 4 target districts Ministry of Health professionals delivering all aspects of Primary Health Care (PHC) services in partnership with NGOs and Village Health Committees “

Definitely a longer-term result, probably at the Outcome or Purpose level:

“Output 3: Selected Central and District Hospitals achieving year on year improvements in national assessments of patient-centred clinical care “ [p. 12]

This is an unnecessarily confusing mixing of real results and completed activities, in the use of one term -- “Outputs”.

According to the DFID revised LFA guide, all DFID projects are now supposed to collect baseline data on indicators for Outputs, Purpose and Goal before approval [p. 33] and to assign to each Output a weight that will “provide a clearer link to how output performance relates to project purpose performance”. This suggests that while not completely responsible for achievement of results at the purpose level, project managers are expected to report on progress at that level and assess the continuing likelihood that Outputs are making a contribution to broader achievement of results.

Of course, the fact that this document is not easily available from DFID leaves open the possibility that the ideas in it may not be in universal favour within DFID.

But there are other documents which suggest that reporting on results, and not just on activities, is important to DFID. The 2010 synthesis review of 970 DFID Project Completion Reports [124 p.] ---which may have laid the groundwork for recent criticisms of DFID performance by the new British government by criticising the lack of assessment of how projects contributed to goals, did note that “The main concern in the [Project Completion Report] process is to assess performance (achievement of the stated purpose),” [p. 36] This is clearly a focus on results, and not just on delivery of products, or completion of activities.

And as an indicator that the focus on results is being taken seriously at the political level, a recently leaked DFID memorandum on which programmes or projects should be cut, suggests that the government should be focusing on projects that can be defended “as outcome focused as possible, and will deliver value for money”, and that DFID “ will only judge ourselves against commitments and outcomes that we assess pass the fitness test.”

As Philip Dearden noted in a discussion about implementation of the new DFID logical framework on the Monitoring and Evaluation News website : “Its very important to remember that many DFID programmes are now spending huge amounts of money and we need to know what changes the money is actually going to bring about.”

The April 2010 DAC OECD peer review on British aid [130 p.] concluded that

“DFID has a strong results-based management framework, and this – combined with a purpose and performance-driven organisational culture and cohesion at the senior level – is important in ensuring effective delivery of the aid programme.”

And, even before the new British Government’s multilateral aid review, there were signs that DFID was concerned about results reporting in multilateral agencies. The 2009 Guide on Using the Revised LFA put it quite clearly:

“…DFID will have to work with the fact that multiple partners mean differences in terminology and approaches.

DFID has played a leading role in ensuring harmonisation of approaches, and is committed to continuing in this vein. However, it is important that in pursuing a harmonisation agenda, we do not relax our requirements for robust monitoring and evaluation tools.

Differences in language and approach should not be an excuse for gaps in information. In fact, the revised logframe format has already been used by DFID teams when negotiating with partners. DFID needs the information in the logframe in order to report to UK taxpayers that funds are being used in the best possible way and delivering measurable results.” [p. 16]

In part because of this, and with the UK review of aid which is about to begin, there can only be more pressure forthcoming on project managers under DfID funding to go beyond reporting on activities and products, and to monitor progress on results.

It will be interesting to see if this applies to multilateral agencies using DFID funds. [Update note, August 13, 2011: The DFID Multilateral Aid Review was completed in March 2011.]

The CIDA Results Framework

Length: 4 documents, 148 p.

Most useful: Results-Base Management at CIDA: A How-To Guide

Getting a clear idea of the CIDA Results-Based Management framework is relatively easy in comparison to some other agencies, as a new CIDA RBM system is being implemented, and most of the definitions are included in 3 documents available online at the CIDA website. The most important of these is the new CIDA RBM Guide. Other, more detailed documents, however, are so far, only available by request from CIDA’s Performance Management Division. This review deals only with those readily available, online, now.

In 2008, CIDA made changes to its results-based management policy, dropping the Log Frame (although not, as has been suggested, the Logical Framework Approach) and replacing it with a Logic Model (a change I will review in more detail in a subsequent post). The former CIDA results chain, in general use up until the end of 2009 looked like this:

Activities→Outputs→Outcomes→Impact,

The new CIDA results chain (just being implemented in 2010) is as follows:

Activities→ Outputs→Immediate Outcomes→Intermediate Outcomes→Ultimate Outcomes.

There have also, as a quick glance at these two will show, been recent changes to the RBM terminology CIDA uses. These are outlined in a useful, although rather densely phrased, policy document produced in 2008, explaining both new and old CIDA RBM terms and definitions [14 p.]. This short document is worth reading because it:

Summarizes OECD/DAC, Canadian Treasury Board and old CIDA definitions for results, activities, Outputs, and Outcomes;

Makes the case for the new definitions; and

Provides very simple examples of a range of possible results chains – both the old and the new CIDA formulations, the OECD/DAC model, and a multilateral project results chain.

In the past CIDA regarded Outputs as near-term results -- for example, changes in understanding arising out of training activities. But, as this document notes “given the almost universally accepted definition of “outputs” by donors, OECD DAC, and [The Canadian Treasury Board Secretariat] as products or services, it is necessary to readjust CIDA’s former term [for near-term results] to “immediate outcome.”

The CIDA change is useful because it makes a very clear distinction between completed activities and results. But it is based on the misconception that other donors in fact do have a clear definition of Outputs as completed activities, and do not permit results to be included in the term “Outputs”. This is something that in practice is clearly not the case for many UN agencies, for SIDA, AusAid, DANIDA, USAID and even for DFID.

The new CIDA Results-Based Management framework, like most other current aid agency documents now defines Outputs as “Direct products or services stemming from the activities of an organization, program, policy or initiative”.

It is worth noting, however, that “changes” are not included in the definition of "Outputs", and one explanation of why Outputs have been redefined as products and completed activities is that the new definition

“Clearly splits development results from products and services (outputs). This distinction should strengthen performance reporting by partners, given that it is now clear they will have to report on both outputs and outcomes” [p. 7 ]

The examples provided in CIDA Results-Based Management publications also make it clear that Outputs are completed activities or products, but not results. For example,

Activity: Build wells

Output: Wells built

Outcome: Increased access to water

Activity: Develop and deliver training on well maintenance

Output: Training on well maintenance developed and delivered

Outcome: Increased ability to maintain wells.

Results in the CIDA context are

“A describable or measurable change in state that is derived from a cause and effect relationship. Results are the same as outcomes and are further qualified as immediate, intermediate, or ultimate. all termed “Outcomes” [ p. 8]

CIDA Immediate Outcomes are near-term results phrased as changes – increases in understanding, skills or access.

CIDA Intermediate Outcomes are mid-term changes, “expected to logically occur” within the life of a project, if Immediate Outcomes are achieved, including things such as increased use of clean water, or improved trust in government.

CIDA Ultimate Outcomes are hoped for long-term changes, the justification for the project, but unlikely to be achieved during the life of a project. These refer to things such as improved health status, or reduced vulnerability of children in conflict areas.

These results categories are illustrated in more detailed documents on CIDA RBM including

“Results-Based Management at CIDA: A How-to Guide" (also available, in slightly different Micorsoft Word format as the CIDA RBM Toolkit [26 p.] , which ties together the Logic Model, indicator development, and risk assessment, and

The “CIDA Logic Model with Definitions” [1 p.] which outlines how Outputs relate to results in the new Logic Model format.

The 2007 OECD DAC peer review of Canada’s aid programme [107 p.] noted that the Results-Based Management and internal audit processes in CIDA were:

“...cumbersome, with limited differentiation in the indicators required and the processes involved for large and small programmes. While this helps to compare results among different activities, efficiency is compromised. The system might also be used to justify risk aversion rather than risk management, especially in those areas where it is more challenging to articulate measurable results (e.g. in governance)” [p. 49]

The OECD/DAC review also noted the relatively strict CIDA application of RBM to small agencies:

“ For example, an application for a small workshop organised by an NGO in Canada has to set out development results as if it were equivalent to a major bilateral programme in a partner country, with requirements to provide an impact evaluation. While providing discipline for NGO proposals may appear reasonable in theory, the practice can appear unnecessarily burdensome to the applicant.” [footnote 37]

The new CIDA RBM system is unlikely to relieve small agencies of the requirement to justify activities in terms of results, but with its simpler approach to visualising results in a Logic Model, it may be an attempt to address the issues raised by the peer review, about the unwieldy nature of the process, and to make the process of identifying results more intuitive.

While the new CIDA results framework is clear and logical, there are, of course, questions about whether it will actually be implemented, not just in plans, but in reports from project managers that are clearer and more useful than those produced before the new framework was put in place. CIDA projects have often had real difficulties in producing baseline data, and it will be interesting to see if the new framework stimulates more attention from CIDA’s own managers and from project directors, to this important element of Results-Based Management.

It will also be interesting to see if CIDA applies its standards on accountability and results reporting to multilateral agencies.

I will discuss how the new CIDA RBM framework, in particular the Logic Model, and Performance Measurement Framework, are being implemented, in a subsequent post.

The EuropeAid Results Frameworks

Length: 10 documents, 710 p.

Most useful: Results Oriented Monitoring Handbook; Glossary of Terms

Despite the obvious inference that the EU aid programme could be treated as a multilateral exercise, for the purpose of this discussion, I am treating EuropeAid as a bilateral agency. But, of course, EuropeAid itself has to account for the multiple results frameworks of its component members, and as the 2007 OECD DAC peer review of the European Union's aid programme [114 p.] noted,

"Because the Community functions both as a donor agency and as a multilateral recipient of Member State funds it is understandable that it does not allocate a large proportion of its funds to other multilateral institutions...." [p. 42]

But because it works with a multitude of member countries, it has to work with differing perspectives on results, and this is reflected in the many different results chains in EuropeAid and related documents, five of which are discussed here.

The EuropeAid site has a few pages which summarize a lot of data. This includes the EuropeAid glossary of terms related to results-based management in which the EuropeAid results chain takes this form:

Activities→ Outputs→Results→Impact

The EuropeAid glossary of RBM terms makes it clear that it neither defines, nor uses, the term “Outcome”. It simply refers to results.

EuropeAid Outputs are defined as “Goods and / or services produced / delivered by the intervention (e.g. rehabilitated road).”

EuropeAid Results are defined as “Initial change attributable to the intervention (e.g. reduced transport time and cost)” leading to

EuropeAid Impacts, defined as “Further long term change attributable to the intervention (e.g. development of trade)”.

The EuropeAid guide on Evaluation Methods [97 p.] provides examples for Outputs such as “teachers trained” or “”advice provided to groups of farmers” and for results, such as “girls benefitting from increased access to education” or “new breeding practices that prevent desertification”.

Both of these examples, among others, indicate a clear difference in this EuropeAid framework between Outputs as completed activities, and results.

But other documents, such as the 2009 EuropeAid Results Oriented Monitoring Handbook [118 p.] , (which is, for some reason, occasionally not available) cloud the picture on how “results” are defined. The Results Oriented Handbook notes, for example that

“Monitors have to fully understand the concepts and terminology used in ROM and to apply them in the correct and coherent manner. This is specially true for ‘efficiency’, ‘effectiveness’, “outcomes’ and ‘outputs’ as these terms might be used differently in other management and M&E systems.” [ p. 48]

The irony is that this document does use some words differently than those defined in the EuropeAid Glossary. While the Glossary refers only to Outputs, Results and Impact, distinguishing between Outputs and Results, the Handbook on Results Oriented Monitoring says of Outputs that they are:

“the goods and services produced; e.g. children vaccinated. In the EC’s Logframe structure these are referred to as ‘results’;” .[p. 29].

The EuropeAid Glossary specifically avoided including Outputs in results, but notes that other EC documents such as the Handbook, may use “the term be used in the wider sense.”

The EuropeAid Results Oriented Monitoring Handbook is, nevertheless, an interesting document, describing a very systematic and detailed framework for monitoring which goes far beyond completion of activities, and includes assessments of Outcomes, even Impacts, and the acquisition of information from stakeholders on relevance, effectiveness and sustainability of results. And within this document there is a distinction between

EuropeAid Outputs as completed activities (eg. Training sessions”),

EuropeAid Outcomes as intermediate results (“improved capacity of those who attended the training”) and

EuropeAid Purpose, or longer-term results, (“improvements in area of intervention due to the improved capacity of the target group”) [p. 65], which are the “specific, central highest ranking objective of the project” [p. 71], the highest level on which a project reports, but not necessarily the highest level on which it is monitored.

EuropeAid Impact as the overarching result to which a project may contribute, and justification for a project.

In answer to the question “How well is the Project achieving its planned results?” the Results Oriented Monitoring Handbook says ‘It is crucial to understand that effectiveness in this part is concerned with outcomes, not with outputs (tangible goods and services).” [p. 71]

There is a clear, and logical results chain here then, and this is reflected in the differentiation between Outputs and changes -- as results -- in the detailed Background Conclusion Sheet [p. 65-74].

Activities→Outputs→Outcomes→Purpose→Impact

The EuropeAid site website’s section on results monitoring , also has 6 very short synthesis reports on results, the most recent of which is for projects in 2007, but none of these documents makes it clear whether the monitoring is focusing on results as changes, or whether this is referring to completed activities or products. There are references to “impacts” but only as potential results, so it is not clear from any of these short reports what “results” are actually being achieved. Presumably the more detailed reports would clarify this.

But also in the past year, EuropeAid has put together, in its Tools and Methods series, two interesting, and potentially very useful documents suggesting that for capacity development EuropeAid is indeed focused on results. These include the March 2009 EuropeAid Guidelines on Making Technical Cooperation More Effective [138 p.], and the EuropeAid Toolkit for Capacity Development [82 p.] Both of these discuss a results chain, for technical cooperation, focused on capacity development in which the attention to results, not just to completed activities or delivery of products is clear. The Toolkit for Capacity Development presents the results chain for capacity development interventions this way [p. 68-70]:

Activities→sector capacity (capacity development outputs)→sector outputs (capacity development Outcomes)→Sector Outcomes→Sector Impacts

In this context as the Guidelines on Making Technical Cooperation More Effective say, “…logical frameworks for “[Capacity Development Technical Cooperation]” need to focus

on outputs and outcomes beyond the immediate deliverables by TC;” [p. 29]

It is clear from the discussions in these two guides that the sector capacity or capacity development outputs are real changes in the ability of host government agencies, for example, to deliver services more effectively. These then, whatever they are called, are results. These Guides themselves refer to the 2007 Guide on Support to Sector Programmes [119 p.]. That Guide uses the simpler results chain of

Activities→Outputs→Outcomes→Impacts,

Outputs in this document are clearly completed activities. [p. 89]

There are other, more complex, results chains referenced in five working papers on results indicators on transport, education, water and sanitation, health and agriculture, all produced in 2009, suggesting a 6-stage results chain:

Activities→Outputs→Outcomes→Specific Impacts→Intermediate Impacts→Global Impacts.

This is focused at the agency level, rather than the project level. These guides too, however, make it clear it is change that must be followed, not just completion of activities.

The European Community's aid volume is huge. As the 2007 OECD DAC peer review noted the "volume of Community ODA alone is larger than that of the World Bank’s International Development Association and several times that of the United Nations Development Programme" [p. 12] and much of it is administered or influence by the work of EuropeAID. EuropeAid should, therefore, have a very big influence on how multilateral aid agencies treat results reporting.

If the EuropeAid Glossary of RBM terms definitions, and the types of capacity results described in the 2009 EuropeAid Tools and Methods guides are used, there is a clear distinction between results and completed activities. It is also clear, despite labelling Outputs as results in the Results Oriented Monitoring Handbook, that the approach described there clearly focuses on, and in theory at least, takes the monitoring of results, not just activities, seriously.

The bottom line: Holding UN agencies to account

Despite anecdotal evidence that they are themselves trying to report on results, given the ambiguity of their own definitions of results SIDA, AusAID, DANIDA would have some difficulty in holding UN agencies to higher RBM reporting standards.

On the other hand, despite some inter-agency differences in terminology, there is now in the RBM frameworks of DFID, and CIDA, in EuropeAid's specific focus on capacity development, and perhaps, if we can trust the definitions on its website, for USAID too, little of the confusion between completed activities and results that persists in the OECD DAC and UN definitions.

There is also in these bilateral frameworks, and despite some criticisms of how they work, little space or excuse for project managers to avoid taking responsibility for reporting not just on completed activities, but on results -- explicitly as those things that have changed after the guides are produced, the training completed, or the schools or health clinics constructed.

Given the inconsistent results reporting by UN agencies described in some of my earlier posts, even those bilateral aid organizations such as DFID, CIDA, EuropeAid or USAID which do, or purport to, hold their own projects to relatively high reporting standards, could be open to criticism for providing funds to UN agencies without requiring from them similar standards on reporting.

In this context, recent moves by DFID to assess multilateral aid could focus attention more clearly on the weak UN results culture, and how bilateral agencies deal with this.

Bilateral Results Frameworks --1: SIDA, AusAid, DANIDA

noreply@blogger.com (Greg Armstrong) — Fri, 30 Jul 2010 19:51:00 +0000

Greg Armstrong --

As bilateral donors consider offloading responsibility for delivering aid programmes to multilateral agencies, it pays to examine what standards these UN agencies use for results reporting. Two earlier posts looked at the relatively weak results reporting in many UN agencies, and questioned why the bilateral agencies don’t hold the UN agencies to higher standards. This third post looks at how three major bilateral donors define their results frameworks.

Level of Difficulty: Moderate to complex
Primarily useful for: Project managers, national partners
Length: 23 documents, 1,337 p.
Most useful: SIDA, AusAid and DANIDA guides to the LFA
Limitations: Mounds of bureaucratic language in many of the bilateral documents make it difficult to identify and take effective guidance from potentially useful material.

Who these materials are for

Project managers, evaluators and national partners trying to understand how SIDA, AusAid and Danida define results and frame their approaches to RBM.

Background – Confusion in UN results definitions

In my previous two posts, I examined how UN Results-Based Management frameworks can provide the rationale for some agencies to avoid reporting on results, and to focus instead simply on describing their completed activities. To a large extent this confusion arises out of the ambiguous terminology used in Results-Based-Management – words that mean different things to different agencies even as they agree to use them in common. This is complicated further in the UN case when agencies apply the same terms to near-term results at the project level as are applied at the aggregated country level. As I noted in the second post, vague definitions for results, even when they are ‘harmonised’, too frequently seem to lead unmotivated or inattentive agency heads and project managers to the path of least resistance: project reporting on completed activities and not results.

This third and a subsequent fourth post in this series briefly survey how the results frameworks of 7 bilateral aid agencies compare to those of the OECD/DAC and UNDAF. This review is only of those agencies where information, in English, could be obtained in reasonable time from their own or associated -- and publicly accessible -- websites. Those already familiar with the discussion of the problems in the UN framework may want to skip directly to the review of the bilateral agencies.

I am assuming, for the moment, a general agreement that “cause and effect” is not a reasonable description of what is intended by a results chain, but rather that it is a notional concept of how activities can contribute to intended results. In this context, this post examines how results chains are presented in documents from the OECD/DAC, SIDA, AusAid, and Danida.

USAID, DFID, CIDA and EuropeAid results frameworks will be analysed in the final post in this series.

OECD DAC RBM

The OECD DAC results chain is a common link between the bilateral and UN agencies, and it looks like this:

Activities→Outputs→Outcomes→Impact

"Outcomes", as used here, compare roughly with what the bilateral agencies work towards as development changes or results. However, it is at the Output level that ambiguity and problems occur. The approved OECD/DAC RBM terminology for Outputs, in French is “Extrants/Produit” and in Spanish it is “Producto”. In both cases, the intention is clear and stated this way, there is nothing obviously wrong with the results chain.

But the definition of (the English) Outputs provided in the OECD/DAC glossary has two elements:

First: “The products, capital goods and services which result from a development intervention;”

and then: “changes resulting from the intervention which are relevant to the achievement of outcomes.”

It is this second clause -- on “changes” -- that blurs the distinction between products or completed activities on the one hand, and results on the other.

And it is this essential lack of clarity in the original OECD/DAC definitions that is now reflected in the UN agency definitions for Outputs. Given that there are two choices here – defining Outputs as completed activities – or as real results (changes) -- and not making a clear distinction between the two, this ambiguity has provided unmotivated UN project directors the excuse to simply report on completed activities.

Bilateral Aid Agency Results Frameworks

Many of the bilateral donors reflect this same ambiguity in how they define Outputs, some specifying them as just completed activities or products, while others include the additional possibility of near-term change.

Nevertheless, whether activities and products are labelled as completed activities or as Outputs, for some bilateral aid agencies, they are not considered sufficient to constitute results reporting in results based management terms.

Thus, for example, an activity that trains 500 health workers may have Outputs (completed activities or products) of “training materials” and 500 “trained workers”. But completion of the training would not be considered a “result” by the tougher bilateral RBM standards.

Instead, most agencies would look for reports of viable results at the primary or near-term level only where it was clear what difference the training made. In this context, the result of the activity would be visible if it could be demonstrated that a substantial number of the 500 health trainees

learned something they did not know before,
changed their attitudes, or
were now working more effectively.

In the longer term, results would presumably be sought also in more systemic types of change --- in changes seen in effectiveness and relevance of the professional development system or in health services delivery policy and practice, and eventually changes in health status for the population.

The SIDA Results Framework

Length: 7 publications, 507 p.

Most useful sections: SIDA Guidebook on the LFA

The 2004 SIDA Summary of the Theory Behind the LFA, [35 p.] which I reviewed several months ago presents the SIDA results chain this way:

Activities→Results/Outputs→Project Objective/Purpose→Development objectives

This LFA guide describes Outputs as “actual, tangible results that are a direct consequence of the project’s activities”. This, on the surface, seems similar to the vague UN definitions of Outputs sometimes being products and at other times low-level results.

While the SIDA guide goes further in saying: “The outputs/results are a description of the value of the services/products produced by the project within the framework of what the project stakeholders can guarantee”, the examples the Guide provides are ambiguous. Some cite completed activities as Outputs e.g.:

Activity: “Repair old water points”; Output: “50% of existing water points…repaired”

On the other hand, other examples cite actual results as Outputs:

Activity: “Train in hygiene”; Output: “Hygienic habits of the target group improved”

This may be why a 2008 SIDA evaluation of Policy Guidance and Results-Based Management for the SIDA education sector [183 p.] noted in its comments on two programmes that:

“Even when civil society or university capacity building outcomes/ outputs are defined in log frames, performance indicators are frequently unclear and the results chain and causal relationship from activity, output and outcome are unclear.” [p.41]

A 2008 evaluation of SIDA Human Rights and Democratic Governance project results [44 p.] noted the ambiguity in how results were reported among projects – some reporting completed activities as results, under the “Output” label, and others reporting on Objectives.

“Training is by far the most frequently mentioned output; it is an output of nearly half of all the projects covered by the sample. Apart from this, the list of outputs includes, but is not limited to: policies, guidelines, studies, publications, information, seminars, study tours, infrastructure, theatre productions, and funding. Some may find that outputs, such as those found in the evaluations and studies on SIDA’s support to HR&DG, are not results but an expression of the different activities initiated with support from Sida.

That is true, but they are, nevertheless, the most tangible and direct manifestations of the interventions. And, as mentioned above, listing these as “results” is in line with SIDA’s and DAC’s usage of the concept.” [p. 11]

This report also noted that

“Rather surprisingly, the study team found that only four of the reports made a clear distinction between outputs and outcomes. In other words, the general picture of the 31 reports in the sample is that they apply this terminology rather imprecisely and give limited room for drawing any conclusions whether projects and programmes differentiate between different result levels.”

This is really not dissimilar to the general thrust of a review of “The Management of Results Information at SIDA” [18 p.] produced in 2001. Since information on

“programme expenditure, activities and outputs are covered in reports on programme progress, SIDA should make sure that counterparts’ annual reports focus on information about programme outcomes and impact.” [p. 15].

The suggested format for this report allocated roughly 80% of reporting space to the results (Outcomes and Impacts) and only about 20% to Outputs.

But although there is a move to strengthen RBM in SIDA, the OECD/DAD 2009 peer review of SIDA [123 p.] noted that “at the time of the peer review visits, many staff remained unclear what results-based management really entails in practice.” [p. 16]

The peer review also noted that

“SIDA’s very detailed manual on managing individual projects and programmes focuses strongly on planning and approval processes but provides no guidance on and makes no mention of results-based management.” [p. 59].

A new guide was planned for 2009, but I could find no sign of it on the SIDA site.

SIDA is not alone in its mixed use of Outputs, of course, and there are suggestions that it may be moving toward greater clarity. A 2007 paper by the SIDA Results Project Group “Strengthening SIDA Management for Development Results” [20 p] suggests that the agency wants reports on results, not just activities. And a 2009 study comparing accountability in four aid agencies [84 p.] found that after criticisms in 2009 by the Swedish Minister of International Cooperation that SIDA was not reporting adequately on results “the first report on Outcomes” was produced in 2009.

“SIDA is now further expected to include these new results data in its annual reports….Meanwhile the “cascading effect” of this results orientation seems to have altered the functioning of SIDA’s major partners responsible for implementing the projects: a number of interviews with these partners notably refer to the growing budget they allocate to measuring projects results.” [p. 56]

One particularly telling suggestion in reference to joint projects and partner capacity for RBM was made in the 2007 Strengthening SIDA Management for Development Results: “In countries where the capacity to generate and report results information is weak, donors should support capacity building” [p. 9]. This is a good suggestion, because projects that build results-based management capacity are rare.

The AusAID Results Framework

Length: 11 documents, 412 p.
Most useful: AusGuide 3.3: The Logical Framework Approach

The AusAID site is not easy to work with, but with some searching it is possible to find three documents -- part of the AusGuide -- relevant to how the agency defines results. The most useful of these is the 2005 AusGuide 3.3: The Logical Framework Approach [37 p.] which illustrates the AusAid results chain as:

Activities→Outputs →Component Objectives/Intermediate Results →Purpose/OutcomeGoal/Impact.

The definition of Outputs here follows the standard OECD/DAC definition as: “… the tangible products (goods and services) produced by undertaking a series of tasks as part of the planned work of the activity” and the examples provided make it clear that these are simply completed activities or products:

“…irrigation systems or water supplies constructed, areas planted/developed, children immunised, buildings or other infrastructure built, policy guidelines produced, and staff effectively trained.”

Only the last, “staff effectively trained”, suggests something approaching a result by suggesting that it is not enough to train, but that there must be some measurement of whether this was done “effectively”. This opens the door to considering change: how well the staff learned something. None of the other examples provided in the guide make it clear that Outputs are more than delivered actions: a training strategy developed; new courses designed; a series of special courses delivered; books produced or distributed.

A further complication is AusAID’s differentiation between “project Outputs” and “contractible Outputs” described in the 2002 version of AusGuidelines on the LFA [45 p] and referenced briefly in the 2005 version. This leaves the impression that project managers may only be responsible for delivery of completed activities.

But another 2005 document, an updated version of its “Promoting Practical Sustainability" [43 p] notes: “Monitoring and reporting frameworks based on tools such as the logical framework approach should look beyond the contracted activity and output levels and incorporate regular assessment of the movement towards achieving sustainable outcomes “ and follows this with some specific recommendations, including that

“The basis of payment should attach payments to outputs or milestones that are largely within the control of the contractor while encouraging the contactor to focus on their contribution to ‘outcomes’."

In theory, at least, this suggests that reports (and the projects themselves) should address much more than the simple completion of activities.

And a 2006 M&E Framework Good Practice Guide [8 p.] distinguishes between Outputs as “products, capital goods and services delivered by a development intervention to direct beneficiaries” on the one hand, and Outcomes as “short-term and medium-term results of an intervention’s outputs.”

More recent publications from AusAid differ in how they address what results are to be reported. The Office Of Development Effectiveness, established to monitor results of projects, does not make reference in its most recently published (2008) Annual Review of Development Effectiveness [74 p] to the results chain found in the AusGuide. It refers instead to the importance of achieving “objectives”, and rating their quality, through the AusAid Quality Reporting System. This apparently involves ratings by “activity” managers of “activity objectives" and “higher level strategy objectives” at entry, and later by independent evaluators, but makes no reference to any standard AusAid results chain or logic model.

Although the website entry for the Quality Reporting System says it “helps to ensure reliable, valid and robust information is available”, in the only two guides to this system that I could find, one an overview of the Quality Reporting System, [6 p.] and the other Interim Guidelines for Preparing Completion Reports [10 p.] there are no references at all to indicators, and two references to “objectives” and “key results” which may or may not be the same thing. In the ODE’s review of development effectiveness similarly, there were only three references to indicators in the context of the Australian aid programme, one of which noted that in support of rather vague “objectives” for Vietnam, “No specific indicators were identified to define the focus or scope of ambition of Australian support meant to meet these objectives.”

The December 2009 AusAid Civil Society Water, Sanitation and Hygiene Fund Guidelines [25 p.] however, clearly identify Outcomes as results, substantive changes such as “increased access to improved sanitation services” or “improved hygiene behaviour”. The same guidelines define Outputs as essentially completed activities, such as “provision of technical support” or “facilitation of dialogue”, [p. 16-17] and provide direction to NGOs applying for funds to describe their approach to “monitoring and evaluation of Outcomes”. [p. 22]

All of this is frustrating for the outsider, because it is clear from the work it was doing on contribution analysis between 2005-2009, that AusAid was genuinely trying to find practical means to document progress towards real results. As one report on the use of contribution analysis in the Fiji Education Sector program [35 p.] in 2007 noted, where previously the Program’s indicators were primarily at the Output level, after using this approach, “the new indicators subsequently provide information on progress towards and contribution to outcomes,…”. [p. 33] This report similarly provides examples of results chains somewhat different from those described in the 2005 AusGuide.

Activities→Outputs→Immediate Outcomes→Intermediate Outcomes→End Outcomes.

This was evidently not a one-shot effort because terms of reference for an AusAid evaluation [12 p.] in May of 2010, made specific reference in describing research methods to “…a basic contribution analysis and counterfactual assessment ahead of the in-country visit.”

The 2009 Peer Review of Australian Development Assistance [117 p.] states that “Performance reporting processes have now been brought together into one coherent system” but it is difficult for the average person, among the vast array of available documents on AusAid and associated websites, to find the coherence.

There probably are guides on the AusAid intranet providing clearer guidance on whether managers should now be reporting on progress towards real results at the Outcome level, but based on what is easily and publicly available, what results AusAid requires of projects, or what data it requires in support of these – whether it expects reports on completed activities and products, or on genuine results -- remains unclear.

The DANIDA Results Framework

Length: 5 documents, 418 p.
Most useful: Guide to the Logical Framework Approach

It is very difficult to get a comprehensive and updated picture of how DANIDA currently understands the results chain. The Danida website has what looks like a very practical and potentially useful guide to the Logical Framework Approach [148 p.] but it was produced in 1996, and it remains to be seen if the results chain defined there as

Activities→Outputs→Immediate Objective→Development Objective

is still valid.

The Danida definition of Outputs in this document: “Outputs are the tangible, specific and direct products of activities which largely are within the control of project management” is, nevertheless, very similar to what the OECD/DAC came up with 6 years later.

The examples provided, such as “Awareness campaign about hygiene conducted”, make it clear that outputs are seen as completed activities and products, not changes. Changes (results), in the Danida results framework, are at the Objective level statements like “Population has adequate hygiene practices”.

Somewhat ambiguously, the suggestion in the document is that project managers are responsible only for Outputs, not for Objectives, while at the same time project evaluations are to assess whether the logic of the project was valid, and whether the Outputs did contribute to the Objectives.

In this sense, the “results chain” expectation is that a project or programme will eventually report on how its Outputs contributed to Objectives. The November 2009, DANIDA Analysis of Programme and Project Completion reports 2007-2008 [76 p.], lists in the annexes the format for these reports, and they include an assessment of

“…the extent to which the programme/component has achieved the general objectives as defined in the partner programme document, and discuss the contribution by Danida to achieving the objectives” .

It is the Project and Programme Managers who must do this reporting, so we would expect that this might generate some interest in collecting data not just on Outputs or completed activities but on the Objectives (results).

But in the Ministry of Foreign Affairs Guidelines for Programme Management [87 p.] updated in 2009, while there are some passing references to Immediate Objectives in the Template for Semi-Annual and Annual Reports [p. 66], the emphasis is clearly on details about Outputs – or completed activities [p. 67-70].

The Ministry of Foreign Affairs Project status report formats for NGOs ask for

“..an account of the project objectives, indicators related to objectives, preliminary outcomes and assessment of the potential of the project for realising the project outcomes(s) established.” .

Indicators in these reports are required for Outputs, but they are also, apparently, and somewhat ambiguously, to refer to Objectives and “Outcomes”.

On the whole, it is not really clear from these sources then, to what extent DANIDA does push for reports on results, rather than completed activities.

It could be that as Denmark decentralizes responsibility for aid management, and aid reporting not just to country offices, but to Danish missions to multilateral agencies, the definitions, and interpretations of what results must be reported on, have become more diffuse.

The 2007 OECD DAC peer review of Denmark’s aid programme [103 p.] suggested that “Denmark could consider further rationalising this reporting system, as it involves many different tools and may be time consuming given embassies’ staffing constraints.” [p. 14]

The bottom line: Holding UN agencies to account

These three agencies, SIDA, AusAid and DANIDA, have a clear interest in increasing the proportion of their aid going through multilateral agencies, and are making real efforts to improve delivery of programmes. Anecdotal evidence is that they take results reporting seriously in practice. But given the ambiguity of their own definitions of results, which often, in different documents from the same agencies, conflict, SIDA, AusAID and DANIDA would have some difficulty in holding UN agencies to higher results reporting standards.

Next post: DFID, CIDA, EuropeAid and USAID

For further reading, and more original documents on SIDA, AusAID, DANIDA and the OECD DAC see:

The Denmark Ministry of Foreign Affairs Aid Management Guidelines
The AusAid publications website
The SIDA publications page (difficult to navigate)
OECD DAC peer reviews of aid agencies

Results-Based Management at the United Nations -- 2: Ambiguous results definitions

noreply@blogger.com (Greg Armstrong) — Mon, 28 Jun 2010 22:07:00 +0000

Greg Armstrong --

Problems with United Nations agency reporting on results can, in part, be attributed to ambiguous definitions of Outputs.

Level of Difficulty: Moderate-complex
Primarily useful for: Anyone trying to understand the UN’s inconsistent RBM system
Coverage: 13 papers, totalling 721 p.
Most useful components: Technical Briefs on Outcomes, Outputs, Indicators and Assumptions
Limitations: Large number of potential documents laden with bureaucratic language

Who this is for

This post, and the previous review of UN agency problems in reporting on results, is intended for bilateral aid agency representatives, national government representatives, project managers, monitors or evaluators, trying to understand the inconsistent reporting of project results by UN agencies. Because the UN documents are often lengthy, and laden with bureaucratic language, it is unlikely to be of interest to people who don’t need to work with UN agency counterparts.

Background: Problems in UN RBM

This is the second of four posts assessing UN agency results chains, results definitions and problems in reporting on results, and (in the third and fourth posts) the results frameworks bilateral aid agencies use. In the previous post, I suggested that inconsistent UN agency results reporting could, in part, be attributable to a weak results culture, and sometimes weak leadership at the country level within some UN agencies. This post reviews how ambiguous results definitions also undermine UN agencies’ credibility in results reporting.

A third post in this series will review results chains for 3 bilateral aid agencies – SIDA, AusAID, and DANIDA -- define results, and the fourth and final post will review how USAID, DFID, CIDA and EuropeAid define results and results chains.

How results are defined in the UN at the country level

Language matters. I have argued elsewhere that the terminology used in Results-Based Management is dysfunctional, largely because the jargon (Outputs, Outcomes, Impact, Objectives, Purpose) can mean many different things and, that in the context of development programming, terms used for results are intended to mean something different than they mean in day to day usage. What works in almost all project contexts, however, is to focus on change as the characteristic of a result. This is a word and a concept that works in most languages, and appeals to people’s desire for common-sense terminology.

For most bilateral donors, whatever the specific terms they use, completed activities -- often referred to as Outputs -- are not sufficient for reporting purposes. While bilateral project managers are obviously required to report on completion of activities, the real emphasis in project reports is expected to be on if and how these activities are contributing to significant changes in the short to mid-term -- changes to knowledge, attitudes, policy or professional practice.

In other words, results.

Some agencies refer to these results as Outputs, Outcomes and Impacts. Others refer to them as Objectives, or Purpose, or as Immediate, Intermediate and Ultimate Outcomes. But whatever the terms, the focus is clear: “Tell us about how the project or programme is contributing to change, not just about how you spent the money”.

The problem for bilateral and national agency partners trying to hold UN partners to reasonable standards of results based management lie, I think, in the vast number of documents dealing with results in the UN context; the ambiguity of the UN definitions of results; and the confusion about how results chains for projects relate to results chains at the country programme level for different agencies.

At the UN Development Assistance Framework level, results from different agencies are essentially being aggregated, and as the 2010 UNDAF guide on reporting [22 p.] makes clear, the UNDAF report should be “focused on reporting results at a strategic level….”. Unfortunately, terms that define results for aggregations of projects at the strategic level do not necessarily work at the individual project level.

So, how does the UN, in the UNDAF and UNDG guides and technical briefs, deal with results?

UN Results Chains

Results chains describe the sequence, and the nature of links between, activities, completed activities, and near-term, mid-term and long-term results. Most practitioners agree that a direct cause and effect between activities and results is unreasonable given the wide range of intervening variables that occur in real life, but that results chains describe the general sequence of how activities can contribute to change – or results.

The several UN documents reviewed here and in the previous post variously refer to results chains (sometimes in the same document) as

Activities→Outputs→Outcomes→Impacts

Activities→Outputs→Agency Outcomes→UNDAF Outcomes→National Priority

Activities→Outputs→Country Programme Outcomes→UNDAF Outcomes→National Priority

Activities→Outputs→Agency Outcomes→Country Programme Outcomes→UNDAF Outcomes→National Priorities

“Activity Results”→Outputs→Outcome→UNDAF Outcome→National Priority

UN Agency Outputs→UNDAF Output→National Outcome→National Goal

Looking at these it is no wonder that there are differences among implementing agencies in how results are explained for projects and programmes in the UN system.

UN Outcomes

Most UN agencies use as the basis for their own definitions of results, the 2003 harmonized UNDG Results-Based Management Terminology [3 p.] which grew out of the OECD/DAC Glossary of Key Terms in Evaluation and Results-Based Management [37 p.]. “Outcomes,” the harmonized terminology states “represent changes in development conditions which occur between the completion of outputs and the achievement of impact.”

Of the hundreds of document available at the UNDG website, the most frequently referenced for elaboration on RBM terms are four technical briefs produced in 2007. These include technical briefs on Outcomes, on Outputs, on Indicators and on Assumptions and Risk.

The 2007 Technical Brief on Outcomes, [7 p.] the apparent foundation for many of the other UNDG documents on Results-Based Management, however, explains Outcomes at the country level this way -- The UN country teams have two separate, but linked, Outcomes at the country level:

UNDAF Outcomes

Country Programme Outcomes -- which incorporate individual UN agency Outcomes. These are not as clearly defined in these documents as are UNDAF Outcomes, but they appear to be seen as the changes to things such as policy or legislation, needed to facilitate long-term institutional or behaviour change. Bilateral donors might see these as mid-level results or Intermediate Outcomes, achievable to some degree over the period of a project.

So, for example, two Country Programme Outcomes of adoption or passage of Human Rights legislation and then adequate budgeting for its implementation might - it is hoped - lead to longer term UNDAF Outcomes of improved human rights in the country.

As the useful checklist in this Technical Brief on Outcomes notes on page 4, a Country Programme Outcome “…is NOT a discreet product or service, but a higher level statement of institutional or behavioural change.” The same checklist adds that a Country Programme Outcome should describe “a change which one or more UN agencies is capable of achieving over a five year period.”

All of this is fairly easily understandable, and as long as the assumptions underlying all of these intended results are monitored, these definitions should open the door for UN agencies to collaborate with other donors and with national governments on solid Results-Based Management, and the reporting of results. Many bilateral aid projects also have a 5-year term so it would be reasonable to see Outcomes occurring in that period.

However, this approach is not always applied, agency-by-agency, to UN results reporting at the project level. The new 2009 UNDP Handbook on RBM, [221 p.] something I will review in more detail in another post, while it has many very useful components, says, of the scope of project evaluations, that the focus should be on:

"...Generally speaking, inputs, activities and outputs (if and how project outputs were delivered within a sector or geographic area and if direct results occurred and can be attributed to the project)* "[p. 135] .

The footnote in that quote acknowledges, however, that some large projects may have Outcomes that could be evaluated. And while later the Handbook says of project reporting that it should include “An analysis of project performance over the reporting period, including outputs produced and, where possible, information on the status of the outcome “ [p. 115] it is clear that at the project level, the priority is on reporting of Outputs.

The UNDP Handbook has a very good discussion of problem identification and stakeholder involvement in the development of a results framework which, it says, “can be particularly helpful at the project level” [p. 53]. But while the UNDP Handbook reiterates the importance of attention to results at the country level, this is less obvious at the UNDP project level:

“Since national outcomes (which require the collective efforts of two or more stakeholders) are most important, planning, monitoring and evaluation processes should focus more on the partnerships, joint programmes, joint monitoring and evaluation and collaborative efforts needed to achieve these higher level results, than on UNDP or agency outputs. This is the approach that is promoted throughout this Handbook.” [p. 14]

The problem with this is that, if attention to results from the component parts of a development programme (i.e. the projects or activities) is missing, and if project results are not properly reported, then the foundation for country-level reporting will be, at best, hypothetical.

On the other hand, a revised draft ILO RBM Guide [34 p.] notes that:

“Some mistakenly think that outputs are ends in themselves, rather than the various means to ends. RBM reminds us to shift our focus away from inputs, activities and outputs—all of which are important in the execution and implementation of work—and place it instead on clearly defined outcomes....” [p. 17]

Confusion Over UN Agency Outputs

It is at the Output level that the real confusion starts and, in turn, I think this undermines attempts to get some UN agencies to think about, or to report clearly on, results at the project level.

The 2003 document UNDG Results-Based Management Terminology -- at least in terms of Outputs -- improved upon the earlier OECD/DAC Glossary of Key Terms in Evaluation and Results-Based Management, when it defined Outputs as “The products and services which result from the completion of activities within a development intervention.”

The original OECD/DAC definition of Outputs was “ The products, capital goods and services which result from a development intervention; may also include changes resulting from the intervention which are relevant to the achievement of outcomes.”

The introduction of the “completion of activities” in the UNDG definitions, and the absence of any mention of “changes” opened the possibility that the UN would have a definition of Outputs that would help it discriminate between activities and products on the one hand (completed training, study tours, texts produced) and results (increased understanding or changed attitudes). This would be compatible with the view of many of the major bilateral donors which see Outputs as completed activities or products – and, while a necessary step in achieving results, not themselves results.

Unfortunately, the 2007 UNDAF Common Country Assessment guidelines [76 p.] complicated the issue by defining Outputs as:

“The specific products, services, or changes in processes resulting from agency cooperation”

In the 2007 Technical Brief on Outputs , [11 p.] there is yet another definition -- and the line between necessary products and the results they contribute to has been further blurred:

“Outputs are deliverables. They normally relate to operational change: changes in skills or abilities, the availability of new products and services. They are the type of results over which managers have a high degree of influence. Failure to deliver outputs is, on the face of it, a failure of the programme or project.”

In the Checklist for validating Outputs, this same document says that “The output is a new product or service, new skill or ability that can be developed and/or delivered by one UN agency working with its partners”.

The definition in the 2009 UNDP Handbook provides some conflicting examples of what Outputs are. In some cases they are clearly completed activities or products:

“Study of environment-poverty linkages completed.
“Police forces and judiciary trained….”

But in other examples, there is the hint of the changes (results) that could come out of completed activities:

“Systems and procedures implemented and competencies developed…” [p. 59]

The last example combines as an Output, a completed activity and a learning result.

The core of the UN RBM problem

The Issues Note on Results Based Management in UNDAFs [11 p.] , analysed a number of results chains and came to the conclusion that they were often illogical, with Outputs more complex and difficult to achieve than supposedly subsequent, and more advanced, Outcomes. Downgrading the complexity of Outputs makes sense, in this context, at the country level, where many results are being aggregated, but when applied to the project level, it may provide an excuse for very limited reporting on results.

And this is the core of the problem.

If we must be limited to the simplistic three-level results chains, and if jargon like “Outputs” must be used (and I am not sure it must), then at the country level, saying that changes in skills or abilities – real development results – are part of the Output, may be legitimate. This may be particularly necessary if we are aggregating results from a large number of projects and trying to fit all of their multiple results into a three-stage description, where the only terms available are Outputs, Outcomes and Impact.

But why combine two different concepts – completed activities and the changes these produce -- under one label? Why not separate the completed activities from the change? The problem of confused labels leading to inadequate reporting, it seems, must be laid at the door of a harmonization process that permitted core RBM definitions to become -- and remain -- intellectually vapid.

When we allow mixing activities and results and labelling them both “Outputs” at the project level, and then tell project managers that their primary reporting responsibility is for Outputs, it provides two potentially dysfunctional things:

A disincentive for timid or uncertain UN agency leadership to take responsibility for achieving anything risky in terms of change -- real results -- focusing instead on the logistics of organizing activities and the delivering products;

An excuse for these same agencies, if they want to, to limit themselves to generating data and reporting on the “product or service” delivery part of the definition of Outputs, rather than on genuine changes –results-- or lack of them.

Neither response is valid, however, even in the context of the UN RBM guidelines. The same UN Technical Brief on Outputs in fact, clearly argues against such timidity -- and this is important for anyone who wants to hold UN agencies to higher standards of project reporting:

“You may be tempted to list things like workshops and seminars as outputs. After all, they are deliverable and some workshops can be strategic if they gather decision takers in one room to build consensus. But, in most cases, workshops and seminars are activities rather than outputs. And remember that outputs are not completed activities – they are the tangible changes in products and services, new skills or abilities that result from the completion of several activities.”

So, the intention was honest, and indeed some UN agencies do make a genuine attempt to report on real and tangible changes or results. ILO, for example, has in its 2005 self-directed learning module on results-based management [131 p], a framework that views Outputs in part as a result. However, with what appears to be a new draft ILO RBM guide developed after 2007, the organization had changed the emphasis. While the definition of Outputs in this new ILO RBM guide is standard, combining activities and some degree of initial change, the document distinguishes more clearly between completed activities and results, and more particularly between Outputs and results.

“RBM is also significant in that it represents a shift away from the narrow focus on inputs, activities and outputs, which are factors internal to organizations. RBM moves the focus outwards towards results, which are the external changes that organizations are working to achieve and the fundamental reason why organizations exist to begin with.” [p. 4]

ILO also has its own version of the country programme results – in its Decent Country Work Programmes – where it insists that reporting at the project level focus on results and not just on completed activities.

“At the ILO, outcomes are developed: at the organization level and appear in the [programme and budget]; at the country level and appear in [country work programmes]; and at the project level and appear in various project reports and evaluations.” [ILO draft RBM Guide p. 7].

Any ambiguity there may have been in the former ILO approach to Outputs has been reduced substantially with the injunction to “Remember that targets are connected to outcomes and indicators, and never to outputs.” [p. 13]

Other agencies, however, persist -- particularly at the project level -- in reporting only on how many people they have trained, how many guides they have produced, how many meetings have been convened. While it might be argued that the UN’s focus is at the country level, and thus Outputs must incorporate some element of change (and not be confined to completed activities), it is precisely at the level of change that project reporting is not, in actual practice, consistently taking place.

The most ironic example of this I have seen was a multi-donor project on aid effectiveness that focused on improving RBM capacity in the national aid coordinating agency. Managed by a UN implementing agency, the project year after year -- and despite bilateral pleas for reports on results -- continued to report on how many people had been trained, how many workshops held, how many reports produced. It never reported on indicator-based evidence of performance management capacity improving, or even that there had been some change in understanding or attitudes towards performance management. Both results could have been demonstrated with little real effort. However, the impression left by its failure to respond to Steering Committee requests for data on change, was that the implementing agency simply was not motivated to report on results.

While UNDP’s policies and procedures for closing a project may say that a final report “should look at sustainability of the results, including the contribution to related outcomes (and the status of these outcomes) and capacity development", it is reasonable to question how this might be done in any reasonable way without the project actually collecting data on Outcome indicators.

Conclusions: Responsibility for results reporting

Project managers may not, in some agencies, be responsible for achieving results (change, not just completed activities) at the Outcome level, but they should be held accountable for managing for, and reporting on, development results – at the very least attempting to influence results. But if we are to take management for development results seriously, and to report on this process, we need to monitor progress by looking at indicators of change – of results -- whatever label they are given.

There is a wide variation in how UN agencies treat Results-Based Management for projects -- with some reporting on actual results of their work with partners, while others limit themselves to the mundane details of how they delivered the activities. This variation might in part be a matter of confusion in the terminology of what they are responsible for. Clarifying this at the Output level, by clearly distinguishing between completed activities, and real results, is a necessary first step in improving the results culture in UN agencies.

Poor implementation of Results-Based Management must also be seen, however, as a matter of leadership. It is the UN agency heads who determine, through what they reward and what they ignore, whether managers will focus on generating and reporting results or settle for just reporting on completed activities. And it is the UN agency country leadership that sets the tone for whether project managers will take results reporting seriously.

Obscure results terminology has just allowed idle or unfocused UN agency leadership to take the path of least resistance, and report on completed activities rather than on results.

The bottom line:

The 2009 bilateral donor assessments of UNDP [46 p.] and UNICEF indicate, as I noted in my previous post, that Bilateral aid agencies’ perceptions of how UN agencies manage for development results are more critical than country partners’ views. But as the bilateral donors provide a huge amount of assistance through these agencies, it would be wise for the UN agencies to take the criticisms of how they manage for results into account. As DFID begins its multilateral spending review, careless results reporting practices may finally come under serious review by the bilateral aid agencies.

Next: The next two posts will look at Bilateral aid agency results chains, as reflected in SIDA, DANIDA, AusAID, USAID, DFID, CIDA and EuropeAid documents.

Results-Based Management at the United Nations – 1: Inconsistent RBM

noreply@blogger.com (Greg Armstrong) — Sun, 30 May 2010 19:49:00 +0000

Greg Armstrong --

There is wide variation in the competence of UN agencies’ use of Results-Based Management. As bilateral aid agencies consider how they can offload responsibility for managing aid programmes to multilateral agencies, it is worth examining how the UN agencies stack up in terms of accountability for results. This is the first of four posts comparing how bilateral and UN aid agencies define results. It reviews publicly available guidelines and policy documents to try to understand the underlying causes for poor results reporting at the UN agencies.

Level of Difficulty of the reviewed documents: Moderate to complex
Primarily useful for: Those trying to understand the UN’s inconsistent RBM system
Coverage: 8 documents reviewed, 546 p.
Most useful: The Draft ILO RBM Guide
Limitations: Dense language in most of the documents, and unresolved ambiguities about what results mean in the UN context.

Who this post is for

This post is intended for bilateral aid agency representatives, host country government agencies, project managers, evaluators and monitors who want to know why there is such a wide variation in the standards applied by different UN agencies to managing for and reporting on results. While most readers will not end up any more satisfied with the UN approaches to RBM after reading these posts, they may understand why there is such variation in performance reporting.

One of the problems in reviewing donor agency policies on results reporting, and sometimes the guides on results based management, is the huge amount of essentially tedious material that the reader must wade through before getting to the heart of what each agency requires, and in particular what it means by “results”. The sites and documents referred to in the first two posts of this series of four, are intended to be of use to people who need, for one reason or another, to make sense of, take guidance from or work within UN agency results frameworks. These posts are likely to be of limited interest to readers who don’t need to worry about UN agency RBM, other than perhaps to gain some insight into why it can be difficult sometimes to pin down “what difference” development assistance is making.

Inconsistent UN application of Results-Based Management

National governments and bilateral donors are often confounded by how some UN agencies appear, on the surface, to resist using results-based management in multi-donor projects, or in projects where bilateral agencies provide grants to a UN agency to act as the implementing agency. Most bilateral donors insist that bilateral projects and programmes track progress on results, reporting not just at the level of completed activities, but for mid-term and even long-term results.

Bilateral donor agencies are held accountable for how they spend taxpayers’ money on bilateral projects. While their results reporting is not always done with complete effectiveness, even in some of the agencies which pay the most attention to results, as recent criticism of DfID reporting shows, the organizational culture of many bilateral agencies at least validates the pressure put on project managers to test assumptions, to collect baseline data, and to report not just on completed project activities, but on progress towards results.

The basic definitions of results – often labelled "Outputs" and "Outcomes" – is roughly the same for both bilateral and UN agencies. The frustration for many of those who are trying to account for how public money is spent on aid projects, however, is that there appear to be different standards for how, on the one hand, NGOs and private sector agencies implementing bilateral projects have to account for results, and on the other hand, the often slack results accountability requirements applied to the bilateral aid that is channelled through UN organizations.

While even relatively small bilateral NGO projects are often expected to demonstrate to bilateral donors some progress in realizing results, these same bilateral agencies often apply different standards to grants made to UN agencies, often requiring and receiving -- even for large projects -- reports dealing in detail only with the completion of project activities.

In one case I can think of, in return for a grant of several million dollars by a bilateral agency to one of the development banks, the progress report -- two pages in length -- described not even the activities undertaken, but simply listed sub-project names.

The report had no details on context, on rationale, or even on activities, and certainly nothing on results.

Meanwhile, several NGOs getting $20,000 from the same bilateral agency were wading laboriously through indicators, collecting baseline data and providing detailed reporting not just on activities, but on results.

Not all UN agencies are alike, of course. FAO and the ILO make a real effort to apply results based approaches creatively and effectively. The draft ILO RBM GUIDE [34 p] for example, is easy to understand, and does put the emphasis on reporting results, not just completed activities.

Other UN agencies such as UNICEF and UNIFEM also sometimes do (in UNIFEM's case "did") report effectively, trying to distinguish between activities and results. But there are still other agencies, such as UNDP, where the approach is being applied -- to put it as gently as possible -- inconsistently. This despite the attempt in 2009, with a new Handbook on Planning, Monitoring and Evaluating for Development Results [221 p], to reinforce a results culture. [Note: by October 2011, this handbook was, at least temporarily, no longer easily available.]

A 52-page 2008 study by Alexander MacKenzie on systemic challenges to the effective use of Results-Based Management at the country level in the UN system reviewed a number of previous studies on RBM-implementation problems and makes the point that while there has been progress in reporting an results at the project level, this is not the case at the country level under the UNDAF.

Some UNDP projects, in fact, do report on results and this may be what the “progress” noted in this report refers to. But it is commonly acknowledged among bilateral agencies collaborating at the project level that -- despite their repeated pleas for reports on results -- many of the UNDP-managed projects to which they contribute millions of dollars are continuing to provide reports that year after year deal only with logistical issues related to completed activities.

The question that needs to be asked is:

Why do some UN aid agencies appear able to get away with inadequate - and in some cases clearly sloppy -- reporting, while in other situations even in the same types of agencies, managers strive valiantly to deliver thorough and frequently highly insightful reports on results?

Assessments of RBM in the UN

There are a number of UN sites with a wide range of guides available. However, the country programming reference guides on results based management at the United Nations Development Group website, and the UNDG RBM working group list of documents and study materials are most frequently referenced in other UN papers as relevant to country-level programming. When I reviewed it for this post, the country programming reference site had 11 substantive papers focusing on how Results-Based Management is being, or should be, implemented within the UN Development Assistance Framework at the country level. Some of those are referenced here, and some in the next post. Documents on the UNDG site relevant to this discussion include:

A UN RBM Action Plan [4 p.] produced in January 2009. This short paper called for a common approach to results based management by UN agencies, common UN RBM training programmes and materials, but not much more.

A number of earlier studies cited in the MacKenzie review [see above] noted many RBM problems at the UNDAF level that will be familiar to bilateral donors trying, even at the project level, to encourage their UN counterparts to comply with some minimal standards for results-based planning, management and reporting:

“Outcomes that are too broad, not strategic, and whose contribution to national priorities is unclear;
Outputs that are not linked to those accountable for them;
Results chains with poor internal logic;
Indicators that don’t help to measure whether results – particularly outcomes – are being achieved and a lack of baselines and targets; and
Poor use and monitoring of risks and assumptions.” [p. 1]

The study’s suggestion that results reporting may be functioning at the project level, but not at the UNDAF level, is a depressing thought given the low level of systematic and coherent reporting on results actually being done at the project level.

Weak UN Results Culture

The 2007 Evaluation of Results-Based Management at UNDP [131 p.] made a crucial point: “In practice, lack of good data in the reporting system is because those responsible for inputting the data don’t see it as something important they are accountable for. It’s just one more imposition from headquarters.”

“In some ways,” as the 2009 UNDP Handbook on Planning, Monitoring and Evaluating for Development Results put it “ it is similar to the difference between having RBM systems and having a culture of results orientation—while it is important to have the systems, it is more important that people understand and appreciate why they are doing the things they are doing and adopt a results-oriented approach in their general behaviour and work.” [p. 12]

The MacKenzie study supports this perspective in noting that, while there had been efforts to coordinate the use of RBM in planning formats, there was little progress on using results information for decision making.

“RBM is understood mainly as planning system – less as a reporting system – and almost not at all as a system to help with day-to-day decisions about programme management. So high quality results information isn’t really needed, or only needed periodically to meet outside reporting requirements.” [p. 7]

RBM systems were viewed by many staff, he noted, as externally imposed things that need feeding….” [p. 14]

Inconsistent UN Agency Leadership in RBM

The 2007 evaluation of RBM at UNDP noted what is, I think, the real cause of inconsistent implementation of Results-Based Management in the UN agencies, in suggesting that the removal in 2002 of mandatory project monitoring tools may have led to a decline in project-level monitoring and evaluation capacity in some country offices, [p. 46 ]. While “… it has stimulated the creation of diverse M&E approaches in others - especially where there is a staff member dedicated to M&E” and “some progress has been made in country offices towards monitoring outcomes”, overall, approaches “fail to explain how projects are contributing to programme outcomes.” [p. ix]

This 2007 evaluation also found that project-level management in UN agencies was overly geared to Outputs (rather than Outcomes), and that although managers have the mandate to adjust operations on the basis of results, “the evaluation found no evidence of results being a significant consideration in that process.” [p. 44].

My point here is that, while variations in approach to RBM are now permitted in the UN system, variations in commitment to results-based management by agency leaders at the country level may in large part explain why, within and among individual agencies such as UNDP or UNICEF, results-based planning and reporting are handled so differently: reasonably well in some countries, but essentially ignored in others.

UNDP can be observed taking project results reporting seriously in one country, but not in another. And even within the same country, UNDAF notwithstanding, it is possible to see UNICEF, for example, treating results seriously, while the UNDP office nearby regards it as, what seems to outsiders, a bothersome afterthought.

Bilateral assessments of UN agency RBM

The 2009 partner and bilateral agency assessments of multilateral aid agencies reflect this difference. On the whole, government partners rate UN agencies more highly on Results-Based Management, than do donor agencies, but donors too see a difference between how, UNICEF, for example, manages for development results (reasonably well) and how they rate UNDP’s Results-Based Management (barely adequate).

The 2009 donor (MOPAN) assessment of UNDP [46 p.], published in 2010, stated that “Donors rate the UNDP as inadequate in ensuring the application of results management across the organisation” [p. 15]. The same document noted that donors “have some reservations about the UNDP institutional culture for supporting a focus on results.”

This is putting it mildly.

In a comparison of the details of answers to specific questions in the donor assessment of UNICEF [47 p.] and UNDP, donors ranked UNICEF higher than UNDP on all five questions related to country focus on results, higher on senior management’s leadership on results management, higher on issues such as whether results frameworks have measurable indicators at Outcome and Output levels, and whether organization-wide strategies have causal links from Outputs through Outcomes to Impacts.

While UNDP rated highly for delegating authority to country offices, this suggests that local leadership on results based management is sometimes lacking. It also suggests that some earlier hopes for improvement in UNDP reporting had not materialized by the time these assessments were produced.

Mitigating Factors

As a colleague and evaluator who has worked as a consultant with many UN agencies explains it, however, the criticisms of some UN agencies’ performance on project-level results reporting may need to be moderated by the realities of context:

UN agency Field staff often face considerable pressure from within the organization to report on activities and expenditures.
Field staff have few professional incentives to take the time necessary for detailed work on indicator-based data collection and reporting at the project level.
Field staff often have to cope with reporting to multiple donors, ranging from major bilateral agencies to small community charities, each with different formats and information expectations.

All of this said, however, where a results framework for a project has been agreed upon between UN agencies, national government agencies and bilateral donors, there can be little to justify not using it.

The bottom line:

United Nations Agencies have an inconsistent record in applying results based management at the project level. Agency leadership at the country level and a generally weak results culture through the system contribute to this, but I think the way results themselves are defined in the UN system also has an effect.

The issue of the UN results chains and results definitions is the focus of my next post.

Using Logic Models in Results-Based Management

noreply@blogger.com (Greg Armstrong) — Wed, 28 Apr 2010 07:16:00 +0000

Greg Armstrong --

This website houses a large number of articles, some of them quite complex, on how to construct logic models, using proprietary software.

The Outcomes Theory Knowledge Base
Level of Difficulty: Complex

Primarily useful for: RBM specialists, academics
Length: 50-60 web pages
Most useful sections: Articles on evaluation

Logic Model review - Greg Armstrong, using Wordle

The “Outcomes Theory Knowledge Base” is the name given to a compilation of more than 50 articles on what the author refers to as Outcomes Theory -- what many of the rest of us refer to as RBM, or management for development results. Most of the articles focus on how to use visual Logic Models for project management.

Who this is for

Readers will need to sift through 50+ articles, all written by Paul Duignan, to find what they need. But, although there is a lot of repetition in many of the articles, some of them could be useful to three groups:

Those interested in learning how a visual approach to results, through the development of logic models or outcome models can clarify results discussions.
Those who want an overview of some broad issues in evaluation.
Those people interested in an academic analysis of how results are viewed in a broad conceptual format.

For most field project managers, and host-country counterparts, the utility of many of the articles will be limited by the relatively dense language used to explain some common-sense ideas (for example, see the article “Problems faced when monitoring and evaluating programs which are themselves assessment systems”. Some simpler summaries on logic model development are also, however, available at a related commercial website, EasyOutcomes.org.

The Utility of a Visual Logic Model

While there is considerable overlap in the ideas discussed in the more than four dozen articles included in what Google refers to as a “Knol” or a “unit of knowledge”, the reader who takes the time to work through these will find some useful material.

By my estimate at least 30 of the articles focus on the advantages to project managers, evaluators and monitors of using a visual approach to managing for results - Outcome Models, Logic Models or other visual representations about the relationship between activities and results. The core of these articles (although each puts these in a slightly different context) is based in some common-sense ideas that many RBM trainers, planners or evaluators may recognise from their own experience. Among these is that in planning, monitoring and evaluating for results, we should:

Focus on results, not activities - and label results as “Outcomes”.
Use a visual logic model to clarify results. This makes it easier to see the relationships between activities and different levels of results than is possible using a Logical Framework.
Distinguish, in the logic model, between a) results and indicators for which an agency is directly responsible in the near term, and b) higher level results, for which attribution to the intervention (for success or failure) is not clear.
Hold managers responsible for two primary tasks: a) Achieving results for which there are clear indicators and a reasonably clear and accepted causal relationship between activities and results; and b) Managing for development results at a higher level, in part by collecting and reporting on indicator data on results for which there is less certainty of attribution.
Frame contracting, monitoring and evaluation within the context of the results, activities and indicators identified in the visual logic model.

Most of these articles also suggest that the proprietary software (DoView) sold through a related website, can help us do all of these things more efficiently and creatively than we can by relying just on tables in word processing software. Taken together these four dozen articles also appear to form a help file for those using that software.

Evaluation issue summaries

At least ten of these articles focus specifically on evaluation. While the author obviously thinks that the visual logic model would assist in focusing evaluation questions, the articles go beyond this, and some provide what could be, for those looking for quick summaries, useful overviews of major evaluation issues.

Among those articles that could be useful to readers, whether they use the author’s software or not, are:

“Impact/outcome evaluation designs and techniques” which discusses in broad terms seven different research approaches to impact and outcome evaluations;
“Impact Evaluation - When it should and should not be used;
“Distinguishing evaluation from other processes (e.g. monitoring, performance management, assessment, quality assurance)”;
“Methods and analysis techniques for information collection”;
"Selecting impact/outcome evaluation designs: a decision-making table and checklist approach", and
"Types of economic evaluation analysis".

Greg Armstrong’s analysis

Key Resources on Evaluation and RBM?

While many of the articles on the logic model and on evaluation are useful, the article on “Key Outcomes, Results Management and evaluation resources” provides fewer useful links than the average reader might expect, from someone of the author’s experience.

The descriptive summary says it contains ”A summary list of key outcome theory related resources for working with outcomes, results management, evaluation, performance management, outcomes-focused contracting and evidence-based practice”.

“ Ahha!” I thought, “just what people who want to learn about RBM should have - ideas from the UN, DfID, CIDA, SIDA, Universities, government agencies, think-tanks, trainers and NGOs.” This could have been very useful to professionals seeking user-friendly tools on evaluation and RBM.

A quick review, however, shows it contains just 14 links - all of them to one of 8 of the author’s own websites, including his blog and twitter feed, and all with links to the sale of the logic modelling software. The author obviously has a history of work in evaluation, and presumably knows of other useful sites.

Links to other relevant sites, such as, for example the Monitoring and Evaluation News, or the Centers for Disease Control's Evaluation Working Group resources would have been helpful to people looking for useful tools.

The list of references to Outcome Theory similarly lists 33 articles, all written by this author. Many of them are probably useful, but a broader net might have brought in ideas about the work other people have done on similar or related topics.

The Value of Logic Models

While there is some overlap in the content of these different articles, the basic point being made here is valid - using a Logic Model diagram as the focus for discussion can, indeed, as I have found recently in workshops in Vietnam, Cambodia, Indonesia and Thailand, clarify differences of perception over results, assumptions about cause and effect, and can energize discussions on project design and evaluation.

In one of the articles on this website, dealing with the value added of evaluation, to governance and policy making, using a visual logic model,
the author writes:

“Outcomes models need to be able to be used in all parts of the decision-making process. In order for them to be able to be used in this way, their visualizations needs to be portable across different media so that they can be used whenever and wherever they need to be used. For example, they should be able to be developed and used in real-time during meetings with high-level stakeholders, printed out in a report, and reproduced on an intranet or the internet. Meeting this criteria requires using appropriate software and laying out an outcomes model in a way that ensures that it is portable”.

Software Limitations for Logic Model Development

I facilitate workshops on developing results frameworks, logic models and indicator assessment several times a year, in almost all cases in countries where English is at best a second language, where internet access is often unstable, and in some cases where electrical power is unreliable.

I have not used the DoView software which many of these articles are linked to, in such workshops but I can see from its description that it could be helpful particularly during facilitation of logic model development workshops. Given that this software was developed particularly with results chains in mind, it could possibly have an advantage over other visual mapping software of a similar nature, such as Xmind, Vue or SmartDraw among many others. Like those other programmes, too, however, the software promoted here has limitations which would diminish its utility for facilitators working in the situations I work in.

At the end of a Logic Model development workshop, one important deliverable is a draft Logic Model and possibly an indicator assessment framework, which the users can take back to their many different offices, in different countries, different provinces or cities, a document they can distribute widely to their own colleagues and their own networks, for further critique and possible alteration.

The price for the DoView software is not high - roughly $35 per copy, cheaper than others that can run to several hundred dollars - but not as cheap obviously, as Vue, Xmind or others that are free. Even the free programmes have a problem with accessibility and portability however. Having used any of these programmes to engage people in a dynamic discussion of results, what do you do next, when they want to continue the discussion with their own partners? Do you ask them all to download and install the programmes?

It is not clear to me that the Logic Model diagrams from any of the visual mapping programmes I have seen can actually be edited with standard, commonly - available word-processing software such as Microsoft Word, OpenOffice Writer, or Google Docs. While Logic Models produced with DoView, Xmind, Vue, SmartDraw and many other similar programmes can be viewed using those programmes, or alternatively in PDF or on the web, and can be pasted into word processing programmes, in most cases they cannot be edited by people who do not have the original software in which the diagrams were originally produced -- making downstream participation very difficult.

For those programmes which are web-based, some editing can be done on the internet, but accessibility does not rest in “the cloud” for people where internet access is not always reliable. The bottom line is that the utility of all of these mapping and diagramming programmes is limited where it is impractical to install specialised programmes on dozens of different computers.

If portability really is the criteria for assessing all of these programmes, then the priority should be not just the ability to view the results in a PDF file or on the internet, but the ability of partners to critique and edit the models.

No Painless Performance Indicators

Another issue, is that none of these programmes, with these limitations, will be easy to link to the other half of the results discussion - in many ways the most time consuming portion of results-based planning -- the assessment of the utility of indicators.

As anyone who has worked through the indicator development process knows, it can take days for project partners, working in groups, to sort through potential indicators, testing them for validity, for the existence of baseline data, for the availability and accessibility of reporting data, for the existence of appropriate research skills, and the time required for data collection and analysis.

While several of the articles in the Outcomes Theory Knowledgebase refer to the tongue-twisting “Non-output attributable intermediate outcome paradox” and make a reasonable point about attribution, none of them makes the job of assessing indicators any easier, any faster or any more accessible for partners.

The Outcomes Theory Knowledgebase web site has many “how to” videos, hosted on YouTube, aimed primarily at helping people use the proprietary software. One of these is titled “Painless Performance Indicators: Using a Visual Approach”. This got my hopes up!

But, foiled again: What the video demonstrates is that if you have already done all of the hard work on indicators, having completed this assessment, you can insert a reference to the existence of the indicator, in the Logic Model, using the software. I am sure this is useful (although it can also be done with word processing programmes and the use of hyperlinks) but the point is that inserting indicators in a visual model is not the painful part of indicator development.

For the time being, until something new develops, I will be sticking with the basic word processing programmes which allow a facilitator to work with participants to develop a logic model (albeit without some of the ease of the mapping software) and then link and integrate it with an indicator assessment worksheet, as indicators are being proposed, tested, rejected, modified and accepted. But, I continue to live in hope, and may revisit the issue of software again later.

The bottom line: "The Outcomes Theory Database" includes articles with some useful arguments in favour of using a visual logic model approach, and some quick summaries of evaluation issues, but there is no magic bullet here.

Other resources on Logic Models:

The University of Wisconsin's detailed and engaging online self-study module Enhancing Program Performance with Logic Models

The W.K. Kellogg Foundation's Logic Model Development Guide.

Using PowerPoint to develop Logic Models, from the Centre for Community-Based Research.

Applying RBM to Policy

noreply@blogger.com (Greg Armstrong) — Mon, 29 Mar 2010 18:52:00 +0000

Greg Armstrong --

Can policy making and policy advice be assessed by the same performance assessment standards and RBM methods that are applied to programmes? Mark Schacter’s views have evolved over the past five years.

Level of Difficulty: Moderate to complex
Primarily useful for: Policy makers, performance-management divisions
Length: 2 papers, 42 pages
Most useful section: What constitutes “good policy advice” p. 7-8 in “The Worth of a Garden”
Limitations: Focused primarily on senior officials, unlikely to be useful to field workers, unless they are in a policy development project.

Who this is for

Senior policy makers and performance management officials in the public service will recognise the issues Mark Schacter raises in all of his writing on RBM. Targeted primarily on Canadian public officials, the issues he raises are relevant to public servants everywhere. The policy discussions, however, are less likely to meet the needs of development field workers, or project managers, unless they are working specifically on policy development projects.

A decade of RBM analysis

Mark Schacter has been working on results-based management, training people on performance measurement and writing about it, since at least 1999. While this is certainly not the only thing he does his writing on RBM has been prolific, and influential. CIDA’s new RBM terminology, formally adopted in 2008, for example, bears a striking resemblance to that used by Mark Schacter in his 2002 paper “Not a toolkit: Practitioner’s Guide to Measuring the Performance of Public Programs”

First at the Institute on Governance, and more recently as a freelance consultant, he has published at least 12 articles since 1998, focused on performance measurement and RBM. Some of these can be found on the Institute on Governance web site, but more are available directly on one of his own web sites: Mark Schacter Consulting.

Should Policy be judged by the same RBM standards as Projects or Programmes?

While the paper “Not a Tool Kit” provides a useful summary of the steps and issues in RBM for the public service, particularly the discussion of tradeoffs on indicators, the focus of this review is on how he treats policy making in the RBM context.

Two of his articles demonstrate how Mark Schacter's views shifted between 2002 and 2006, although I think, only marginally, on the important issue of whether standard performance measurement processes can -- or should -- be applied to policy development and public servants’ role in providing advice.

Is Policy Unique?

A strong advocate of intelligent application of RBM to the management of public programmes, in his 2002 article, What Will Be, Will Be The Challenge of Applying Results-based Thinking to Policy, he reviewed the standard arguments against applying results-based management to policy -- that policies are intangible things, subject to a variety of influences, such as politicians’ short-term political needs, that there is often a huge lag in time between policy development and any chance of seeing concrete results. The conclusions, for those who take this view -- and I have heard this recently -- is that policy is therefore “unique”, and that tracing the effect of advice on the success of policy is therefore in some way, “unfair”.

Schacter took the view, in this 2002 article, that intangibility and complexity are not unique to policy but occur often in programme implementation. He appeared to take the view that while these things present challenges to people attempting to assess performance, they also provide an opportunity to use performance measurement, and the critical examination of a logic model, to clarify assumptions and test the understanding of the intended results, implied or clearly stated by policy.

Evaluation and Performance Measurement

The link between the views Mark Schacter held in 2002, and those he expressed in 2006, is the role of evaluation in assessing the effectiveness of policy. Performance measurement, he wrote in 2002, looks at where we are today, and tries to assess how likely we are to achieve long-term results, by looking for evidence that we are making progress against shorter-term results.

Evaluation, on the other hand, assesses not just whether results have been achieved, but whether they were the most appropriate results, why results were, or were not achieved, and whether alternative means of achieving them would have been more appropriate. [p. 17]

The case for performance measurement of policy

Performance measurement, he wrote in 2002, has its limitations, particularly given the usual lag between policy development and achievement of long-term results. But, he continued:

“Sometimes a less-than-perfect instrument is, under the circumstances, the best one for the job at hand. Performance measurement is indeed a “second-best” instrument – but a very useful instrument nonetheless….Citizens have no less a right to be informed about the performance of policies than of programs. In order to explain and justify the allocation of resources to …any policy (or program) you need to have a way of connecting what you are doing now with where you want to be in the long term. This connection needs to be clear and must make sense not only in the minds of the people responsible for the policy, but also in the minds of external stakeholders (citizens, civic groups, private sector operators, politicians, etc.).

Performance measurement helps you make that connection. It helps you tell a believable and compelling story about why a policy was conceived in the first place, and whether or not it appears to be on the right track.” [p. 24-25]

The case for evaluation of policy

By 2006, in a paper for Canada’s Treasury Board, “The Worth of a Garden: Performance Measurement and Policy Advice in the Public Service” Mark Schacter had apparently come to the conclusion that measuring policy performance in the short term, might in fact be too challenging, and that an emphasis could probably be more productively put on longer-term evaluation.

He outlined two options for using performance measurement on a regular basis to assess progress towards policy results. The first is to assess what he called the “process and outputs standards for policy advice”, the second is essentially what he advocated in his 2002 article - to assess progress toward achievement of immediate and intermediate outcomes - whether policy advice was accepted and implemented. The conclusion he came to in 2006, however, differed from his earlier view:

“Low-specificity organizations and tasks pose especially difficult problems for performance measurement – problems so significant that it may be impractical (if not impossible) to apply standard performance measurement in a way that yields useful results. This does not mean that one should not attempt to assess the quality of a policy shop’s performance. But it does suggest that evaluation may be worth considering as a better tool than performance measurement for this particular task. Evaluation, though closely related to performance measurement, differs from it in ways that may provide a better fit with the subtleties and ambiguities of the policy-advice process.” [p. 11]

Greg Armstrong’s comments:

Are performance assessment and evaluation mutually exclusive?

At no point did Mark Schacter advocate abandoning the assessment of policy units’ performance. He has always maintained that at some point policy functions have to be assessed.

What is unclear to me, however, is why the assessment options -- regular performance assessment and eventual evaluation -- appeared to be regarded as mutually exclusive.

It seems to me that combining a) an assessment of the quality of policy advice, and the processes which lead to it, with b) an assessment of interim results, and c) a longer-term evaluation, is a reasonable (if obviously not perfect) way of helping policy advisors, policy makers, legislators, and those who fund them, to understand the progress they are making toward long term results.

By 2008, Schacter was writing about other performance assessment issues, and in How Good is Your Government: Assessing the Quality of Public Management [2008] policy was mentioned only once, in passing. One type of information he proposed in that article, for assessing the efficient management of resources, however, was “Results-based performance information is used routinely as a basis for continuous improvement of program/policy performance.” [ p. 5]

This suggests that he had not given up completely on the contribution to the policy function of regular performance assessment.

How RBM applies to Policy Projects in International Aid

It is important to note that none of Mark Schacter’s writing, at least between 2002-2006, was focused on whether performance assessment could be applied to improved capacity to provide policy advice.

If, as I contend, there is a role for performance assessment in assessing progress on policy in general, there is surely a much clearer role for it, and for RBM in general, in planning, implementing and assessing results for international aid projects which focus on the development of capacity for policy research, policy formulation, and legislative capacity.

Mark Schacter maintained in 2006 that the provision of policy advice is essentially an output - a completed activity. But while there could be a case made that this is true for some policy functions, in the context for which he was writing -- and even that is not completely clear to me -- it is not true for policy capacity development. Improved quality of the policy making process, and improved quality of the advice provided, are both clearly interim results in capacity development terms, and therefore worth assessing on a regular basis.

In the 2006 paper, Schacter outlined the commonly regarded criteria for assessing the quality of the policy advice process, adopted in part from studies in Australia and New Zealand:

The timeliness of the advice for decision-makers
Relevance of the analysis to the current realities faced by decision-makers
Stakeholder consultation underlying the proposed policy
Clarity of purpose (essentially - does the policy itself rest on a solid logic model)
Quality of evidence, and the link between evidence, policy and purpose
Balanced range of alternative and viewpoints reflected in the analysis
Presentation of a range of viable options
Clarity in presentation
Pragmatic assessment of the potential problems of implementing the policy.

All of these, with some work, could form the basis for useful performance indicators for policy capacity projects or programmes and have, in many cases, been used for this purpose. Certainly as Mark Schacter observed, indicators relevant to these issues would provide qualitative data -- subjective in nature, and time consuming to collect.

But, in my experience, qualitative data are not necessarily any more time consuming to collect than quantitative data, and certainly not less valid if the intention is to assess the quality of the policy formulation process.

The bottom line:

Policy development is, indeed, sometimes an uncertain process, but there are ways of improving it, of building capacity and of assessing this capacity. Mark Schacter’s articles on the role of performance assessment in policy clearly outline the challenges, but also deliver some reasonable suggestions on how to deal with them.

Further reading:

For a useful discussion of the indicator tradeoffs between accuracy and practicality, see Not a “Tool Kit”: Practitioner’s Guide to Measuring the Performance of Public Programs, [2002] pages 25-27 and particularly the figures 8-11, p.41-45.
For a summary of the arguments against, and for using RBM at all in the public service, see Trashing Results-Based Management or Throwing Out the Baby with the Bath Water [2006].
For an attempt to create a performance assessment framework to be applied to whole-of government performance, see How Good is Your Government? Assessing the Quality of Public Management [2008].

Indicators: a simple analysis

noreply@blogger.com (Greg Armstrong) — Wed, 10 Feb 2010 04:54:00 +0000

Greg Armstrong --

The Ants and the Cockroach: A challenge to the use of indicators, and A Pot of Chicken Soup, A response
by Chris Whitehouse and Thomas Winderl; and Rick Davies’ commentary

Aesop’s Fables explains indicators. A deceptively simple and engaging introduction to some complex arguments on indicator development and use. Simple metaphors outline the case both against, and for the use of indicators in results-based management.

Level of Difficulty: Simple to moderate
Primarily useful for: Anyone who wants to review the basic arguments on indicators
Length: 12 pages, combined
Most useful sections: Engaging metaphors for RBM
Limitations: Too short to guide practice

Who this is for

Anyone who wants an introduction to what the past ten years of debate on indicators and RBM are about, or anyone who is fatigued with more sophisticated analyses and just wants a laugh. Trainers might find this useful too when the usual RBM exercises fade.

Simplifying the indicator debate

Development professionals know how complex the process of developing good indicators can be. Doing it effectively - or at least in a manner that will give you a fighting chance of producing information relevant to how a project is performing - requires the major participants in project planning and implementation to work together. Exploring the validity of an indicator, its political implications, and the practicality of data collection, can take a long time and can be, as evaluators say, challenging.

It is not that the process of defining usable indicators is necessarily intellectually difficult, but it does require sustained attention. There are so many questions that need to be answered as you weed out the useless indicators, that the process invariably takes time, and patience.

While answering them may be time consuming, the basic questions that need to be asked in assessing indicators, are simple, something that often gets lost in more complex analyses.

In 2004, Chris Whitehouse, who was a United Nations Volunteer Programme Officer in Bhutan, took on his colleague, Thomas Winderl, who was the UNDP’s Head of the Poverty Unit in Bhutan, in what looks like a good-natured debate on the utility of indicators. The result is this short combination of two points of view, apparently simplified to the very basics. I say “apparently” because a careful reading of the articles reveals the complexities the authors were obviously aware of, when they produced these papers. This particular article appeared on the Monitoring and Evaluation News website, in 2004, and the exchange was later included in an anthology of articles on monitoring and evaluation, published in 2006, titled: Why did the Chicken Cross the Road

Other critiques of the LFA and RBM

The core of the arguments against indicators and the LFA were not particularly new at the time. The article appeared several years after Rick Davies and Jess Dart had begun working on an alternative to the use of the LFA, at about the time they were writing the Most Significant Change Guide. http://www.mande.co.uk/docs/MSCGuide.pdf The Outcome Mapping Approach had also been developed largely by IDRC, roughly 3 years earlier, and was also starting to gain some momentum. This was also at a time when interest was just beginning to appear in what is now referred to as “impact evaluation” or more confusingly as “counterfactual” analysis.

The debate on the use or misuse of indicators, the LFA and RBM, has continued since 2004, but never in such a user-friendly format as was the exchange between Chris Whitehouse and Thomas Winderl.

Challenging the validity of indicators

Chris Whitehouse introduces what he sees as some basic issues in indicator development --

whether the indicators are measurable,
who will collect the information,
cost and baseline data.

But the potentially dry discussion is made more engaging through the metaphor he uses - the stakeholders are millions of ants, the project involves first, the movement of a dead cockroach to their nest, and second, after learning lessons from that experience, moving a dead beetle to the nest. The project manager is the Queen ant, persuaded, perhaps against her better judgment, to measure the effectiveness of individual and group progress on getting the cockroach to its destination.

Assigning 10% of the work force to monitoring, a host of different indicators are tested, data collection becomes chaotic, the workers are distracted from their task, focus obsessively on indicator data collection, and decisions that will achieve the indicator, but ignore the underlying long-term result - more food. At the end of the whole sorry process, nobody knows anything useful about the result, but they have learned a lesson about the perils of using indicators.

Interspersed with the saga of the cockroach is the less engaging, but more realistic discussion of the same issues as they might appear in a UNDP project focused on providing computer training for civil servants. Together the two stories make the points that are now familiar to anyone who has been following attempts to make the logical framework approach more usable, or to develop alternatives to the approach:

Technically valid indicators are often impossible to measure
Measurable indicators are often trivial or misleading
Good indicators can take so much time and money to develop, that they interfere with more important work
Focusing on indicators can bias programming decisions to the exclusion of more useful activities that might not produce measurable indicator data.
The use of indicators implies cause and effect, a scientific validation for activities - and without control groups, this is not valid.

Chris Whitehouse’s conclusion is that the logical framework approach is primarily useful as a tool to test the logic and assumptions implicit in project design, but the use of indicators misrepresents and ultimately undermines that simple utility.

Defending the use of reasonable indicators

Thomas Winderl replied with the analogies of cooking soup, and trying to figure out what the weather is. What indicators do we have that will tell us when the soup is ready to eat? What indicators tell us whether we should wear an overcoat or take an umbrella on a walk?

The soup is relatively easy - watch it, does it boil? Is it too hot to eat? Too cold? No need for complex equipment, or outside expert monitors here. Similarly, if we look out the window we will have some fairly useful indicators about the weather? Is it snowing, raining? What are people wearing? Shorts? Gloves?

The points he makes are, again familiar, but well summarised here:

Money spent on indicator development and monitoring should be proportional to the overall project -- but in fact most project vastly underallocate resources to monitoring and evaluation.
Monitoring bad or poorly developed indicators is a waste of time but putting reasonable time and money on indicator development and monitoring is a good investment.
Indicators will skew management decisions in an unhelpful manner only if the indicators are irrelevant to the longer-term goals of the project (Outcomes or Impacts, depending on which agency’s jargon is involved).
The process of developing indicators is itself a valuable part of the design process, helping - or forcing - participants to be clear with each other about what they mean when they talk about results.
The arguments against bad indicators skewing behaviour are an argument for spending time to develop good indicators.

Measurement and verification

Finally, there is a third short and simple paper, produced separately at roughly the same time, that addresses these issues. Rick Davies responded to Chris Whitehouse’ Ants and the Cockroach, with a one and a half-page commentary in September 2009. His primary points, as I see them, are:

No single indicator is ever likely to capture the process of change, but that multiple indicators might contribute differently to our understanding.
Obsession with measurement is a problem - but while objectively verifiable indicators do not necessarily have to be measureable, they must be verifiable.
Control groups are not the only means of attributing results to projects. Most large projects have enough internal variation that with a little thought differences in results can be compared to differences in the way assistance was rendered.

Conclusions about indicators, results logic and assumptions

I agree with Thomas Winderl, and Rick Davies that indicator problems can usually be solved - if enough time and attention are paid to them.

But for me the most interesting part of Rick Davies’ response is in how he deals with the horizontal and vertical logic of a project. He notes that within the context of the Logical Framework, indicators are usually critiqued in terms of their external validity: do they measure what they are supposed to measure?

But, he adds, the more important question is whether the whole change process is actually clear: Do inputs, assumptions, completed activities, results and indicators hang together in a reasonable way? Questioning this internal validity of project design is more important, and deserves more time than it gets now. “We need more attention to theory, and perhaps a little less obsessing about measurement…. And a verified theory holds the potential of replicating successful processes of change elsewhere.”

In my experience, the serious examination of assumptions: taking the time to clarify -- and then to monitor --assumptions about the development problem, assumptions about our theory of development, and assumptions about the working situation necessary for a reasonable chance of success, remains in practice one of the great unexplored areas of project implementation in a results-based management context. If few resources are spent on developing and monitoring indicators, even fewer are allocated to the most fundamental of all issues in project and programme design: examination of our assumptions.

Without serious attention to, and examination of the multiple dimensions of the assumptions we are working with over the lifetime of a project, we are unlikely to learn lessons we can use in future practice, we are unlikely to identify underlying misunderstandings that can undermine our current work, and we are unlikely to spend public money in a responsible and effective manner.

The bottom line: None of these three very brief articles will walk you through the sometimes complex process of finding a good indicator, but they do point out the main arguments that have arisen over indicators, and which have been expressed in often more complex articles or books over the past 15 years.

Further reading on indicators and results

If this was too simple for you, there are more complex, yet interesting assessments of indicators and of alternative approaches to reporting and evaluation of results:

Education for All, The Quality Imperative and the Problem of Pedagogy by Robin Alexander examines in detail the failings of current commonly accepted indicators for education results.

A Political Economy of Aid Effectiveness, by James Morton critiques major approaches to evaluation, and the disincentives for genuinely useful evaluations.

The Network of Networks on Impact Evaluation has a number of publications on how, if you want to, conduct impact assessments.

CIDA’s Practical Guide to Planning Large Development Projects

noreply@blogger.com (Greg Armstrong) — Mon, 25 Jan 2010 21:48:00 +0000

Greg Armstrong --

A Results Approach to Developing the Implementation Plan: A Guide for CIDA Partners and Implementing Agencies

Download it before it disappears. Peter Bracegirdle’s Guide to developing the Project Implementation Plan is a logical, carefully structured tool for planning large projects. It can be useful, not just for people working on CIDA projects but for anyone trying to put together the pieces of a complex project or programme.

Level of difficulty: Moderate
Primarily useful for: Planning large projects
Length: 98 pages
Most useful sections: Examples illustrating how each component of project planning works.
Limitations: Some CIDA-specific RBM jargon and it takes considerable time to work through these processes.

CIDA’s former comprehensive project planning Guide

This document is nine years old, and although some of the terminology in it has been superseded by new RBM terms CIDA adopted in 2009, this is still one of the most useful guides to planning large projects - for any donor - currently available.

But, it is just barely available. It is no longer directly downloadable from the CIDA web site, and it is not available even from sites such as the Monitoring and Evaluation News, which have most, even “out of print” donor documents. A Google search for the Guide returns 577 hits, but most are bibliographic references by other donors, providing a link to a CIDA site, that itself has expired. In theory you might be able to get a copy from CIDA by sending an email to CIDA’s Performance Management Division but this is hardly the apex of accessibility.

I have an old PDF copy of this guide, and a couple of hard copies, but the only place that I found where you can actually download the file (at least in January 2010) is at the website of the author, Peter Bracegirdle’s Appian Consulting.

CIDA is revamping its results-based management terms and guides, and the document may have been removed from the CIDA website pending adaptation to the new terminology, but I would not count on the PIP guide, as it is known, being reintroduced soon.

Who this Guide is for

This is not a guide I would suggest for small projects. It is long and will take time to work through, time that may not be a good investment if a project is short-term or low-cost. But for anybody starting out on the planning process for a longer (2 years or more) complex, or expensive project, regardless of who is funding it, this guide can help put the sometimes daunting process of building and then fielding a complex project into a usable, coherent framework.

While the guide has some CIDA-specific jargon, the logical and (everything is relative) user-friendly structure of the guide can help produce a functional, and understandable plan for most large projects.

This was one of the most important of a series of documents on results-based management produced by and for CIDA roughly ten years ago. The people most likely to benefit from using the PIP guide, therefore, are obviously those working with CIDA projects. Some of the terms are a bit dated: CIDA has changed from the pure LFA approach, for example, now calls its results “immediate, intermediate and ultimate Outcomes”, uses a logic model instead of a results chain, and has a more complex risk identification process. But the logic in the Guide is still sound and could easily be adapted to current CIDA usage or, for that matter, to the work of other agencies.

This guide has considerable potential utility for anybody trying to turn a general project design into a functioning on-the-ground project, for people who want to, or have to, burrow down into the details of planning. It is a tool that should be used with a group, or in multiple sessions, with key project implementation staff, partners and for some components, with stakeholders.

I avoided using this guide for the first several years after it was published, simply because it was so long, and, I assumed, too complex for the projects I was working on. That is still true where the projects are under $500,000, but for large projects I was wrong.

When I did use it in a planning workshop on a justice project with a UN agency that was not required to use the CIDA format, we found that while working through the process took considerable time, it was not really intellectually difficult. The Guide helped focus discussions, and tease out the logical implications of how we were putting the project together. I have since used it on two other projects, including governance and environment projects and I regret not using it earlier.

Format: Quick Notes on Project Planning

This is not an RBM guide by itself. It assumes that readers have at least an introduction to results-based management. But it takes the logical elements of RBM -- problem, results, resources and activities -- and ties them together, something like one of those plasticised templates used to explain Word, or EXCEL or some complex database programme.

Each of the 22 units, in six categories, is covered in a two or three pages, including one or two paragraphs on key concepts, a list of clear questions to focus group work, and a practical example of how the questions and framework can be applied in a project.

There are, in total, 111 questions that can be used to focus discussion. The first three units are specific to the CIDA project implementation plan process in particular, but there are, in the remaining 18 sections, probably at least 80 or 90 questions that could usefully be examined for any project.

This is not deep, not revolutionary, but it focuses attention on what we need to know if we are planning a project likely to have any practical impact.

This guide assumes the basic project design (the conceptualization of the general need and direction) has been completed, and that now the reader is tasked with doing something to bring ideas into implementation.

How Long will it take to use it?

My experience has been that with a relatively small group of people -- ten key staff planning a two-year, single-country project, worth about $2 million -- at least a week is required to work through, in a very cursory manner, the issues outlined here. That built the foundation for the project, but much more work still had to be done to nail down the baseline data, and flesh out the details of the operational and reporting tasks. A project of that size would probably require, therefore, at least a month of full time work to do this properly. Unit 5 of this guide, for example, has ten key questions about the development problem, and examining them carefully with key partners, could easily take two or three days.

On a larger project ($10 million over 5 years) with multiple partners, three weeks of work was not sufficient to cover all of the territory, but the basic structure of the project, implementable --although requiring much followup -- was in place after going through the process. For a project of that size, another two months of serious concerted attention would be needed to get baseline data for the indicators, and through this process eliminate the impractical indicators.

Resistance to spending money on project planning

Many donors and implementing agencies do not encourage “long” planning periods, but this is ultimately short-sighted. A reluctance to spending time and money on rigorous planning just means double the time later on monitoring and remedial design.

In CIDA’s projects, for example, while a project implementation planning “mission” to the field before project implementation might be scheduled for a month, it can often take up to a year (or more) for the actual plan to be approved. This is because, in large part, insufficient time and money is often budgeted or spent at the beginning, examining the logic of the project, the way the logic relates to operation, and, at its most basic, focusing on collection of baseline data.

When it later becomes clear that indicators have not been tested, and the logic of the management structure of funding arrangements is vague or confusing, budget and programming delays, and endless rewrites of the plan often are the result.

It does not seem unreasonable to me that for $2 million, a month or two should be spent on serious planning, and two or three weeks each year on reporting. And for $10 million or more, three or four months of serious attention at the beginning, and a month a year later for internal monitoring, can save months of wasted time and perhaps millions of dollars on unfocused activities. OK, it may cost $200,000 to put together a competent plan for a $10 million project, but what is that -- 2% of the total project cost, to focus the effective spending, and reporting on the other 98%?

There are in some cases, examples of projects costing well over $50 million, where no clear attempt to examine the logic was ever undertaken, until critical evaluations raised the uncomfortable questions that should have been asked many years, and many millions of dollars before.

For consultants who do both planning and monitoring, donors skimping on planning should not be a problem -- because they will get the work later, anyway, as donors and executing agencies scramble to retrieve the mess created by the rush to implementation.

Key Questions for Effective Results-Based Project Planning

The first three chapters introduce where the Project Implementation Plan is in the CIDA structure, and those not working with CIDA can probably safely skim these.
The other major sections of the guide include units, with key questions and examples on

Assessing information requirements for practical planning.
Defining the Development Problem.
Clarifying the logical framework
Reach and beneficiaries
Risk analysis
Incorporating cross-cutting themes, such as gender or the environment
Sustainability strategies
Defining a management structure
Clarifying partner roles and responsibilities
Specifying oversight processes
Relating results to activities and work tasks (the work breakdown structure)
Using scheduling to focus attention on assumptions behind activities and results
Relating results to budgets
Developing internal monitoring, risk management and communication processes.

None of these is exciting, but in the process of working through each, some very interesting ideas about results, assumptions and processes can emerge from a workshop.

Limitations: This is a checklist. It does not explore any of these ideas in detail, and genuinely working through these issues will take considerable time.

The bottom line: This guide won’t do the work for you, and it won’t implement the project, but it will help you define a rational structure for a large or complex development project. You may decide to use only part of it, but there is a real logic to the sequence here. Taking each part seriously and working it through, does, it turns out, make sense.

The LFA Debate: Making the LFA participatory

noreply@blogger.com (Greg Armstrong) — Mon, 11 Jan 2010 22:46:00 +0000

by Greg Armstrong

SIDA's LFA Papers:

3. The Logical Framework Approach - an Appreciative Approach

This is not a complete guide to participatory use of the LFA, but it could help focus attention, in the initial needs assessment stage of project development, not just on problems, but on opportunities. Much of what is said here was old news in adult education 80 years ago, but good ideas do occasionally merit repeating.

Level of difficulty: Moderate
Primarily useful for: NGOs looking for participatory needs assessment techniques
Length: 24 pages
Most useful section: Guidelines on running a workshop, p. 12-20.
Limitations: Academic jargon may distract the reader, no discussion of indicators or reporting

The LFA Debate (part 3)

Prepared by the SIDA Civil Society Center and published in 2006, this is the third in a series of publications produced by SIDA dealing with the different perspectives on how and whether, to use the Logical Framework Approach in development programming.

The first document of the three I have reviewed was a summary of the basic elements of the LFA, published in 2004. The second was a discussion of how civil society organizations critiqued the use of the LFA.

Who this document is for:

This document is rooted in the experience of a Swedish Civil Society Organization (Swedish Pentecostal Mission’s development cooperation agency) working in Africa, and is apparently aimed at project designers and planner working in NGOs. The guidelines for conducting a workshop, and the suggested time allocations for different components, suggest the document is aimed at organizations with fairly small projects.

Overview - Useful, but not new

This document proposes that “Appreciative Inquiry” an approach to research which focuses on how participants perceive their own strengths, problems and opportunities be applied to the Logical Framework Approach used by most development agencies. The model presented here focuses on planning, not evaluation, and is based in part on field trials in Niger, Nicaragua and Tanzania in 2005.

While the ideas have some utility, the use of academic jargon to label what is essentially participatory planning, is likely discourage from reading it, some people who might otherwise find some reasonable ideas here.

Not that the ideas are necessarily new. Much of what is said here was old news in modern adult education 80 years ago: Start with the learners, build on their strengths, help them explore their own potential. John Dewey, Edward Lindeman, Moses Coady, and Roby Kidd among many others have been making the same case, give or take a few academic flourishes, for the past 100 years. But good ideas do occasionally merit repeating.

Principles of Appreciative Inquiry

The basic premises of Appreciative Inquiry, as summarised in Appendix 2 of this short document, appear to be that:

1. The language we use affects our perceptions of ourselves and our communities. “By changing our language, e.g. by talking about opportunities and strengths instead of weaknesses and threats, we can alter our mental frame of reference, and thus our reality.”

2. The process of inquiring about a situation (needs assessments, for example) affect the situation itself:

"Change begins the moment we start to ask questions and study someone’s experiences and perceptions, and the questions we ask determine what we will find.”

[This is described in more general approaches to qualitative research as an “interaction affect”, something with which anthropologists and most other professionals applying qualitative research methods will be familiar.]

3. The way people talk about their situation, is a narrative, and interventions - research and action -- should take particular note of what people say as the basis for understanding.

4. The expectations people have about the future -- their “vision” or preconceptions -- affect the actions they take, and contribute to, or bias that future:

“Studying our preconceptions and expectations about the future, and formulating desirable visions of the future on the basis thereof, will help us to take positive, action-oriented steps in our lives.”

5. Building a positive attitude in discussions is important for change:

“Positive, affirmative premises are needed to build and maintain the forces of change at a deeper level. The more appreciative our starting premise, the more successful and sustainable our efforts to bring about meaningful change and development.

Applying Appreciative Inquiry to Project Planning

In applying these rather general concepts to project planning, this document is quite specific about the context in which it suggests appreciative inquiry can be useful, particularly in reference to SIDA guidelines on working with civil society organizations:

1. The views of the poorest people should influence decision making about what will affect them (presumably including development projects)

2. There should, at the end of the intervention or series of interventions, be long-term improvements in the situation of the participants

3. The capacity of local groups and civil society organizations should be strengthened by interventions.

The Logical Framework approach, in theory, although often not in practice, begins with stakeholders identifying a problem and its causes, then specifying results which would presumably improve the situation. And, taken in this context, there is really nothing surprising in what this document proposes to do with the basic steps in the Logical Framework Approach.

This paper was intended as an integration of what are basically participant-centred methods with the Logical Framework Approach, so it is reasonable that the basic steps of the LFA would be incorporated, but adapted slightly. And, where the concepts might appear a bit vague, in the text, the document presents, in Appendix 1, a 9-page outline for a workshop. This guide lays down a set of prescriptions about how to use this approach - detailed enough to specify group size and even the amount of time in hours and minutes to be allocated to each stage.

Steps in the Appreciative LFA process

This document breaks the usual LFA process into nine steps, and although there are some slight variations in sequence or how the steps are combined, they are not significantly different from what the SIDA 2004 LFA summary paper described, from what Philip Dearden described in his 2005 guide to using the LFA, or from what most agencies now use. My comments, if any, are noted in brackets.

Step 1 - Identification of necessary participants [One component of stakeholder analysis in other LFA formats]

Step 2 - Situation assessment: A Description of the current situation -- both what works well, and what does not work, or works poorly.

Step 3 - Analysis of what the consequences are of this situation -- undesirable effects, and any useful effects that participants do not want to endanger or undermine through programming. In this stage too, the document includes a discussion of what the participants would like to change.

[Steps 2 and 3 together are sometimes in the usual LFA process addressed in problem identification - although again, this “appreciative inquiry approach” emphasises identification of opportunities as well as problems]

Step 4 - Analysis of the factors affecting the situation [This often happens in assessment of the causes of problems in other LFA approaches - and is frequently included in the problem identification stage. Here, again, this document emphasises also analysis of the causes of successes, not simply the causes of problems]

In this stage, several questions are asked:

What things, people or processes are working well - supporting positive aspects of the situation, or mitigating negative aspects of the situation?
What factors, similarly, are contributing to negative or undesirable effects of the situation?
What is the relationship between these factors - do they interact?
Which of these factors - the ones leading to positive or negative results, should a project or programme concentrate on?

Step 5 - Analysis of assumptions and internal conditions necessary for success. Here two questions are asked:

Based on the analysis of factors, which groups are most important to change?
What strengths and resources do these groups bring with them, and how can they be effectively applied?

[It is clear - and refreshing to see - that this paper takes the discussion of assumptions seriously. Accounting for assumptions is implicit in most LFA formats, but rarely directly addressed in practice. In the accompanying workshop guide, in Appendix 1, two of the nine pages are given to descriptions of how to work through a discussion of assumptions in the group]

Step 6 - Identification of the project goal and “deliveries” (deliverables).

Based on the analysis of problems, strengths and critical factors affecting both, what changes should the group focus on?

[This looks like a mid-term result - achievable by the end of the intervention -- what OECD-DAC defines as a medium term effect, or Outcome -- essentially, however ,just a longer-term change to which the project hopes to contribute.]

What will the project “deliver” to contribute to this change? :

The example provided is:

“Local decision-makers and leaders in civil society organisations (CSOs) have been educated on, and understand children’s rights”

[This seems to combine, what in other circumstances would be Outputs - completed activities --- “decision-makers…have been educated….” with short term results -- ‘…decision makers….understand children’s rights.”

c) What strengths - identified in step 4, during the analysis of factors -- can different participants bring to these activities and results?

d) What are different groups prepared to take responsibility for?

e) What are they key factors we need to consider to get results? [This seems, although it is unclear, to correspond to the risk assessment or risk analysis in other LFA terminology]

Step 7 - Overall Goal of the Project

To what longer-term result is the project likely to contribute?

[This clearly seems the equivalent of the OECD/DAC “Development Objective”, CIDA’s “Ultimate Outcome”, or in more basic terms simply the long-term change to which the project is intended to contribute.]

Step 8 - Resources and division of labour

What resources can different participants including the donor bring to the activity?
What organizational capacity needs to be improved?

Step 9 - Action Plan

[This, combined with step 8 seems to correspond to the activity development or activity design component of other LFA processes.]

What concrete needs do we have; how will labour be divided; what are the deadlines, and methods of reporting?

Greg Armstrong’s Comments:

1. Adult Education and Project Planning

The overall idea of putting more emphasis on strengths and not just problems seems to me to be constructive, but it is not particularly new, as I noted in the introduction to this review. Putting the label “appreciative inquiry” on the process of starting with strengths, does not necessarily make it innovative, but nor does it invalidate the utility of starting with how people see their situation, problems and opportunities.

What putting this academic label on the process could well do, however, is discourage field-oriented practitioners from reading this, and therefore from learning something useful, because nobody wants to get bogged down in academic jargon. Results-based management itself has enough jargon without adding a new layer of academic terminology.

If academics really insist on applying their considerable expertise to practical issues, they would be well advised to reduce the jargon. But, that may be an unrealistic hope. if they don’t eliminate the jargon, however, what can we expect next:

Transformative Learning and the LFA?

Critical Self-reflection and the LFA?

Reflective Dialogue and the LFA?

Conscientization and the LFA?

The Dialogical LFA?

2. Needs Assessments and RBM

Needs assessments have always, in theory, started with an attempt to understand the strengths and problems of individuals, communities, institutions or countries, and then have attempted to assess available but underused resources, to build on strengths and minimise problems.

The appreciative inquiry approach explained in this document makes that process of assessing strengths and weaknesses explicit, and that is probably useful, because over time, there has been a tendency to focus on problems. Analysing what works, and trying to build that into a programme is essentially what adult education does - building on our strengths to overcome our problems.

3. Sequencing Results and Activities

The general sequence suggested here -- at least on the identification of problems, strengths and how they play into a potential definition of needs and the structuring of a project, is reasonable, but not revolutionary. But steps 6-9 are somewhat problematic for me. It would seem to me that focusing on the broader change - the long term result (step 7) might logically precede the definition of the immediate project results (step 6), which might eventually contribute to a desirable long-term change.

The danger of focusing first on project results is that short term changes might not really be relevant to a long-term need. Some universities, for example ,have in the past been notorious for assuming that the result of every project should be more graduates. While that might serve the university’s immediate need, (particularly in terms of job security for professors) it will not always contribute to the solution of a longer-term development problem.

Limitations: Indicators and reporting

This document is intended as a discussion of planning, not implementation or reporting. So, it might be unfair to criticise it for not discussing reporting. In fact, it would probably be fair, analysing the guidelines for the workshop, to say that approach may be aimed at improving primarily the LFA stage of situation analysis, not the concrete activity development needed to finalize a project. If limited to this stage in the development of a project, I think it could be said that the document makes some useful contribution to the process of defining needs.

The paper does at least suggest, however, that it is integrating the participatory process not just with needs assessment, but with the whole Logical Framework process, and if the LFA, or results-based management in general are to have any utility for aid agencies, reporting has to be addressed.

If a participatory approach to planning is used, the implications of this for reporting need to be discussed in detail if the participatory approach is to have any credibility with field workers or donor agencies.

Indicators - the evidence that will tell us if we are making progress on results, are an essential (and often criticised) element of practical results-based management. Although the word “indicator” is mentioned 54 times in this document, the complex process of identifying practical indicators is never actually discussed. There is no discussion of what the group process of identifying an indicator might look like, no discussion of the potential problems such an approach might present in practice, and particularly strange, coming from advocates of a participatory approach, no discussion of the strengths of group identification of indicators,

The bottom line: This report provides a reminder of the utility of engaging stakeholders in real discussions about problems and strengths of their community, during the project planning process. Although in practical terms there is little here that is startlingly new, those interested primarily in focusing on the situation analysis, needs assessment and problem identification stage of RBM and the LFA process could probably make use of these approaches. As a guide to the whole LFA process, however, it is unlikely to be as useful as the 2004 SIDA publication summarizing the LFA process..

Because of its limited scope this paper does not resolve the debate on the utility of the LFA. Of the three papers prepared for SIDA between 2004-2006 on the LFA, it is probably the first, on the theory behind the LFA that is likely to be of the most practical use to field-based development practitioners.

Results-Based Management Websites

Online Results-Based Management Training: The University of Wisconsin's Logic Model development course

Format: Interactive Online learning and a downloadable PDF

Course Contents: Reinforcing important lessons about RBM

What is a Logic Model?

More about Outcomes

More About Program Logic

What does a Logic Model Look Like?

How do I draw a Logic Model?

5 different approaches to Logic Model development

How Good is my Logic Model?

Using Logic Models in Evaluation - Indicators and Measures

Glossaries and references on logic model development, RBM and evaluation

Podcasts 3: BBC's More or Less: Behind the Stats

BBC Podcasts

Roughly 9 Thousand available BBC podcasts

More or Less: Behind the Stats

Respect for Data

120 Available episodes of More Or Less

A wide range of indicators

Using More or Less as a research tool: What the data said about 2010

Indicator data on crime and social change

Indicators on defence spending

Misleading indicators on immigration

Compared to what? Risk indicators

Hans Rosling on the quality of indicator data

Cash as an indicator of bank viability

How Incomplete Data reporting undermines indicator utility

Further Reading, viewing and listening on indicators and results

Podcasts and RBM - 2: Audio Podcasts from the BBC, Australian Broadcasting Corporation and National Public Radio

Results-relevant Podcasts from the BBC

Results-relevant Podcasts from the Australian Broadcasting Corporation (ABC Radio)

Hiding the pigs…

A Results-Relevant Programme from National Public Radio

Podcasts and RBM 1: How to use audio podcasts to reinvigorate thinking on indicators and results

Podcasts provide an escape from the drudgery of reading about RBM

The drama of results and indicator discussions

Podcast length

Difficulties in referencing or sharing podcast data

The Mechanics of accessing and listening to podcasts

Time limits on available programmes

What topics are available?

26 lessons about RBM from the 1990's remain valid today

Lessons learned about RBM in the last century remain relevant in 2010

Developing Performance Measurement Systems

Using Performance Information

Further reading on Lessons learned about RBM

Reporting on complex results: A short comment

Are Aid Agency Requirements for Reporting on Complex Results Unreasonable?

Reporting on Complex Change

Further reading on complexity, results and evaluation:

Bilateral Results Frameworks --2: USAID, DIFD, CIDA and EuropeAid

Background: Ambiguous results chains at the UN, and some bilateral agencies.

The USAID Results Framework

The DFID Results Framework

The CIDA Results Framework

The EuropeAid Results Frameworks

The bottom line: Holding UN agencies to account

Further reading on Results-Based Management at DFID, USAID, CIDA and EuropeAid :

Bilateral Results Frameworks --1: SIDA, AusAid, DANIDA

OECD DAC RBM

Bilateral Aid Agency Results Frameworks

The SIDA Results Framework

The AusAID Results Framework

The DANIDA Results Framework

Results-Based Management at the United Nations -- 2: Ambiguous results definitions

How results are defined in the UN at the country level

UN Results Chains

UN Outcomes

Confusion Over UN Agency Outputs

The core of the UN RBM problem

Conclusions: Responsibility for results reporting

Further reading on UN agency Results-Based Management

Results-Based Management at the United Nations – 1: Inconsistent RBM

Inconsistent UN application of Results-Based Management

Assessments of RBM in the UN

Weak UN Results Culture

Inconsistent UN Agency Leadership in RBM

Bilateral assessments of UN agency RBM

Further reading on UN Results-Based Management issues