This note prepared by Rick Davies, documents the results of an opportunistic analysis of data that became available as a result of the Open Space sessions on Wednesday 14^th May. Open Space discussions are different in that participants in discussion groups are allowed and encouraged to move between groups whenever they want to, at any time during the group sessions, not only during the group sessions.

Network Analysis

Such interventions are based on the premise that improving governance, particularly at the level of service planning and delivery, will lead to improvements in services and in the lives of community members.

This document contains a summary of the discussion that was held around this topic in the Round Table on Impact Evaluation held by IRC in Delft, The Netherlands the 13 & 14 of May 2008.
Round Table on Impact Evaluation – Final report

Block 2: Methods

• Qualitative Information Systems – Deirdre Casella
• Outcome Mapping – Harry Jones
• Networks Models & Impact Evaluation – Rick davies
• Accesing the Impact of Participatory Development Programmes – Ratna Reddy
• Dichotomy between Global and Local Monitoring – Kristof Bostoen

Block 3 – Indicators:

• Identifying Indicators example from the agriculture sector – Alistair Sutherland
• Measuring the Impact of School WASH- Rick Rheigans

After more than 20 hours together and a lot of discussion, a lot of questions were still open.

After the presentations on Indicators, the group went into an Open Space exercise. The dynamics of this exercise is based on four basic principles:

1.       Whoever comes is the right person

2.       Whatever happens is what have to happen

3.       When it is over it is over

4.       Whenever it happens is the right moment

Those principles are supported by the two feet law – this law says that when you feel that you need to move on, you should do so.

The objective was to look at urgent issues identified through the discussions, the inputs from the Online Discussion and the issues that each person is interested in. This was a way of allowing participants to focus down on their key areas of interest.

The dynamics of the exercise was:

·         All the participants of the Round Table had the opportunity to suggest a topic for discussion by presenting it to the group and posting it, defining the time and place for discussion. The person who suggested a topic acted as a leader of the discussion. Each discussion leader was responsible for producing a report at the end of the discussion. Those reports will be made available in the weblog when we have gathered them all together.

·         Once the topics were posted, all members of the group were able to choose which discussion they wanted to be involved in

·         Nine discussion were held between 11:00 and 15:00

The discussion topics were:

·         Indicators for Governance

·         Impact assessment as a process making data live after the event

·         What follow up (joint activities, knowledge, products, etc). Would you like to see from this event?

·         Impact Evaluation for a district level WASH governance intervention (Design + Details) where we are developing a methodology + change for “Good Governance”

·         WASH impact as part of Long Term multi-stakeholder sustainability assessment/ reporting

·         Corruption and Governance towards outcomes for impact

·         How do we manage for results (outcomes/ impacts) towards institutional change plus Good Governance

·         Measuring partnerships for service (R&D) delivery

·         What are the essential questions we should be asking regarding WASH Governance to achieve Impact that is sustainable and scalable?

CLOSING

After the open space exercise, the groups met again in order to reflect on this exercise in particular and the Round Table in general.

Online Discussion

Finally Internet connection was available, thus it was possible to look at the weblog directly from the site J

The objective is to continue using the weblog as a platform for sharing materials and information related to this topic.

Participants will e-mail links and interesting information to Sandra Segura. All participants are invited to give feedback on the summaries of the Round Table blocks and the report that will be produced  (target date – by the end of May).

The weblog will be available at least until the IRC Thematic Overview Paper is published on this topic.

Feedback

·         In general the group was satisfied with the Round Table structure and the time for discussion. The open space exercise was highly appreciated

·         There is the general feeling that it should not be the end of the Round Table, it should be the start of a good discussion

·         The group was satisfied with the opportunity for sharing and learning. The combination of skills, background and sectors facilitated this process.

·         There was the feeling that a certain moment the discussion was too abstract, although the open space exercise facilitates more concrete discussion

In closing this report; on behalf of IRC I would like to thank you all of you for your Inputs and motivation during this event J

Blog reports have been prepared by Sandra Segura with support from Peter McIntyre

Sun & Music… good combination for starting the reflection on yesterday’s session.

The round table began the second day with two presentations about defining indicators for institutional change. By chance, both presentations focused on projects in Kenya.

All presentations will be available at the General Information Box

PRESENTATIONS

Presentation 1: Richard Rheinghans – Emory University, Atlanta

Measuring success of school WASH: evaluating impacts or evaluating for impact?

The team at Emory University is working with CARE and other partners in Western Kenya to address the question of whether schools could be a key entry point for WASH even though children of school age are not a key target group (i.e. they do not have the highest rates of mortality.  This is an ongoing project. The hypothesis was that this is plausible because:

•             Schools have an institutional presence in the community

•             In Kenya there is relatively equitable access

•             Children can influence parents. Could schools trigger behaviour change?

Research and Evaluation Objectives

The aim is to assess the strengths of children’s influence in motivating households to adopt health-seeking and hygiene behaviour: to assess the potential for sustaining and scaling school based WASH interventions.

Research interventions were jointly planned with CARE and partners (who carried out the interventions, starting in 2007). Emory was responsible for the evaluation. The aim was to determine:

•             Improved knowledge and uptake of behaviours like handwashing

•             Diffusion of behaviour change into the community

•             Health impact anaemia, helminths and diarrhoea

•             Educational impact (absenteeism and test scores)

•             Cost effectiveness

Long-term outcomes (for such things as diarrhoeal mortality) cannot be directly measured and need to be extrapolated. The team is seeking ways to extrapolate for example from improved test scores to (eventually) to increased productivity.

Emory conducted:

School-based data collection in a total of 185 schools in three different districts. In each school: 

·         There was an assessment of facilities

·         25 children were interviewed

·         Water quality was tested

·         Hand rinse sampling was conducted (to measure contamination)

Community assessment through

·         Household interviews

·         Helminths and anaemia testing

Measures of success depend on the questions you want to answer. Questions include: 

·         Do school programmes provide valuable benefits to children and communities?

·         Is this where money should be going?

·         Did this programme provide benefits to this population?

·         How can this programme provide the most benefits?

Each of these questions demands different tools to find an answer.

Donors recently asked a series of tough questions about effectiveness.

•             How can we ensure we are doing the most we can?

•             Are interventions likely to be sustained after three years?

•             How can we maximise the sustainability of the interventions.

How to measure this became a touchy question. In general, in projects, nobody wants an evaluation because performance is never quite what you promised. It seems a risk because evaluation can lead to the cutting off of funds.

In early 2008, Emory conducted a survey in 55 of 60 schools where CARE had implemented a minimum package of interventions in 2005.

They looked at some key indicators to show sustainability. Did the schools

·         Provide water in safe storage containers?

·         Test drinking water?

·         Provide soap and water for hand washing?

·         Self report containers and taps that were broken.

The results in fact showed rather low sustainability:

·         36% of schools providing drinking water,

·         20% had treated water but only 9% had chlorine residual (which showed that water had been treated).

·         18% provided water for hand washing but only 2% (one school) had soap.

In some ways this confirmed their worst fears about sustainability. However there were some more encouraging indicators:

·         75% had repurchased water treating products.

·         50% had allocated funds for water treatment and soap.

·         16% had active health clubs

It also identified barriers to lack of water access:

·         Hardware design

·         Financial resources

·         People and institutions

An important learning point was that they sent time with partners to identify strategies to address these barriers. They were then able to make a more positive presentation to donors: Here is what we were doing. Here is where we are missing the mark. Here is what we are planning to do to close the gap.

Rick Rheingans addressed how this approach could pose questions related to Governance evaluation.

·         How can improvements in governance provide measurable benefits to users?

·         Did the project provide measurable impact to beneficiaries?

·         Did the project provide improvement in governance or likely to do so?

The bigger question is whether there an evidence base to connect project demonstrations of improvements in governance or outcomes to long term impacts?

Discussion 1

·         Going from outputs to outcomes creates risks for projects and incentives are needed to do that. The risk is that you will be ‘punished’ for failure – instead you need to find incentives to make it more attractive to the NGO or implementer.

·         There was a control group of schools that will receive an intervention later and they were randomly selected.

·         Some of the barriers did relate to governance. One of the improvements they needed was more support from the Ministry of Education.

·         Other key issues were leadership within the school and community support.

·         This was outcome evaluation: what would impact evaluation mean in this context?

·         Impact evaluation would be the educational behaviour and health effects to different interventions such as anaemia, diarrhoea etc. We measure what we can measure and maybe can extrapolate a lot of things we really care about like productivity. We think about impact as the magnitude, and sustainability as the time period. The research would assign an economic value to the cost of diarrhoea. However, the impact on mortality (life years) would have to be modelled because we cannot measure it directly.

·         Governance issues become more important as you move from a project to scale. Then the point of entry is no longer the school, but the government (Ministry of Health or Education or Water). Co-ordinating groups become the intermediaries between those and the NGOs. Responsibility for different interventions, lies with different agencies.

Presentation 2:  Alistair Sutherland  - National Resources Institute UK.

IDENTIFYING INDICATORS:  AGRICULTURE R&D….

This presentation focuses on presenting an example from the agriculture sector in Kenya on the considerations and methods used for defining the indicators related to an action research project.

The objective of the project was to define how it could be possible to show the impact of conducting a research. The focus of this project was to look at the good practice that facilitates to put the findings of a specific research in use.

The most relevant aspects covered during the research were:

·         Local service extension delivery: The objective was to look at the impact or importance of using the results of the research from the service delivery perspective. Among others factors such as coverage and how the good performance of the service deliverer is rewarded were analysed.

·         The definition of targets and its application was also mentioned. The targets defined in terms of service delivery are not always realistic.

During the design phase, there were some delays due to that it was not clear at the first place how the results of the study would be used.

The expected outputs of the research were defined in four main groups. Group 4 was related to influence policy. The objective was to cover all the aspects that can influence the use of research results.

In order to conduct this research, some indicators were defined. The following are the highlighted points related to the methodological approach and aspects considered when defining those indicators:

·         The starting point was the project Log-frame

·         They used a mixture of indicators. Nonetheless, the focus was to answer WHY – Qualitative information

·         The involvement of different stakeholders at different levels was considered important

·         Different perspectives were considered when defining the indicators

·         The KAP (Knowledge – Attitudes and Practice) model – model for defining change was used

·         It was pointed out that the use of simple indicators (e.g age) could be meaningful in some cases

·         Other less “SMART” indicators can add value (e.g. Economic indicators and indicators developed by the community)

·         Indicators were defining considering the perceptions and needs of the beneficiaries. Often they are focused on the tangible results, it is what is important for them

·         The definition of indicators should be done considering how it could be analyzed and presented to its users

·         The context around indicators is also important. Some indicators are much more meaningful when information about its context is available and presented

During the presentation some insights on how it is possible to engage Policy makers were presented. One of the most important actions mentioned was to allow beneficiaries to talk about/ present the results of the indicators. In this context the definition of good governance was discussed. There was not a final word, but it was clear that this definition has an impact on the way in which a specific project is designed.

At the end a summary of points for reflection was presented. The highlighted points were:

·         Tension between using IE as a learning tool, while it is not defined and done but the implementers

·          Can corruption be defined as an outcome / impact?

·         Can Governance be defined as a process? Can we change it?

·         Is the intervention theory of projects focused on Governance issues clear?

 Discussion 2

After the presentation, the following points were discussed:

·         How to balance the need of having quick lead indicators and to define indicators related to IE? – The discussion brought this dilemma; the answer is that what is important is to be clear about what can be measured and which moment. Also what is clear is that there is the need of having different kind of indicators in place (Outcome, input and outcomes), it facilitates to say something about the project performance

·         An example of using mix methods was mentioned. It is possible to add value by looking at both; the outcomes (quantitative) and changes/ achievement at the institutional level.

·         Risk management should be considered as important factor during the whole project life cycle. It has been found that after a long period some project are “deleted”, because the risks around it were not considered

Your answer will be discussed during the block 4 of the Round Table.
The deadline for submitting comments is May 14th 2008 at 15:10 – Central Europe Summer Time

For more information about this Online discussion on Impact Evaluation, please go to the right hand side of this weblog – General Information Box, or visit http://www.irc.nl/page/40054

This is the last Online Discussion question. We want to thank you for your participation during this event

Your comments added high value to the discussion and will be used for producing a Thematic Overview Paper on Impact Evaluation.

Your comments on the last report from the Round Table and the report will be appreciated.

The afternoon session was focused on methods. The objective was to have the opportunity of discussing a sample of methods which consider different variables and approaches (quantitative/ qualitative)

Just so that you can feel our pain, KPN the national telecom provider in the Netherlands had a major crash leaving us in this wonderful venue without Internet access. The ‘near time’ blog was therefore serviced by Sandra – who wrote the blog in the first place – jumping on her bicycle and finding an alternative system so that she can upload and download. This gives a whole new meaning to “posting” a blog.

We started by taking a quick look at five methods that are available for evaluation. The presenters each had 12 minute to present what could each be a whole da’s discussion – so these are very much summaries. Their compete PPTs will be made available on the General Information Box. At the end of the afternoon there was an opportunity to question each of the presenters. Beneath each presentation is therefore a brief summary of some of the points from that discussion.

Picture by Peter MCIntyre – Denis Zoungrana from 2IE Burkina Faso and Sompit Puyaratabandhu from the Office of Agricultural Economics in Thailand voting on what they see as the main challenges in evaluating WASH governance.

PRESENTATIONS

Presentation 1

Kristof Bostoen from the London School of Hygiene and Tropical Medicine (LSTM) focused on what he calls the dichotomy between global and local monitoring. He says that people often expect to get from local monitoring what they get from global monitoring. This is a mistake that often leads people to claim to measure impact when they don’t.

He gave a brief history of monitoring water and sanitation outcomes in the sector to underline that people have been struggling with this question for a long time. During the water decade (1980’s) WHO was the only organisation monitoring sector progress, based on data from national authorities. They filled in a questionnaire that did not take into account what was collectable or available, and they used different definitions for ‘access’.

In 1990, WHO and Unicef joined forces in the Joint Monitoring Protocol but continued to rely on national data, until 2000 when the LSHTM collated the Global Water Supply and Sanitation Assessment report, almost entirely based on population based data. From then on the reports have used national surveys but support to local sector monitoring disappeared.

Survey questions were open to interpretation – e.g. what is you main source of drinking water? Does time to collect water include waiting time? Analysis is often weakened by the fact that neither questions nor answers can be compared between countries. Sampling is also a problem (who you sample and how).

JMP is very focused on MDG targets and so does not example address sanitation (for which there is no target). There are also technical requirements that lead for example to previous estimate values changing for each publication.

Global and national data needs are different. It should be possible to collect water at a local (rather than national) level that would still be compatible with global data needs. At global level the JMP does not do a bad job (better than some critics say) but the nearer you get to local level the less useful it becomes.

The main lesson to draw is that there is no universal indicator. You need a lot of different data just to decide on one aspect – e.g. access.

Points from the discussion

· There was a way of refining questionnaires to make data compatible between local and national level.

· It would often be much lower cost for country bodies to do monitoring themselves because they did not needd to cmpress their activities into such a short time frame.

· The questionairres prepared for JMP could be improved on. However, if someone else produced a quesitonnaire it would not be accepted, becaus the bodies would nto be considered as authoritative. Once people start editign the quesitons they end up asking something different and the comparability is lost,

· Very often in governance we look at the impact of the infrastructure. People value that pipe or toilet rather than the dialogue and the software approaches are forgotten.

Presentation 2:

Harry Jones from ODI:

Focus of Outcome Mapping

Outcome mapping starts from an understanding that focusing on measuring impact is not realistic in relation to many development projects. The ‘impact’ of a particular intervention, especially policy, will not be felt properly by beneficiaries for a number of years. We need to consider the limitations on the influence of a project, and shape our planning, learning, and accountability functions around outcomes, which are further ‘upstream’ from impacts.

A project has:

· Sphere of control – its operational environment

· Sphere of Influence – Relationships & Interactions

· Sphere of Interest – social, economical, environmental states & trends

·

We cannot control everything we’d like to see change. Sustainable change involves the combination of a number of different factors, and is a product of the interaction of many different actors and stakeholders. The more successful a programme is, in some ways, the sooner its influence disappears (because activities and methods are integrated). We need to need to recognise messy realities.

Outcome Mapping is therefore concerned with the level where a programme has direct influence. The focus for learning and accountability are the boundary partners who are individuals, groups, or organisations with whom the programme:

· Interacts with directly to effect change

· Anticipates opportunities for influence

· Engages in mutual learning

You look for changes in their behaviour that would contribute to the wider change you would like to see. Behaviour is defined as:

- Actions

- Activities

- Interactions

- Relationships

In monitoring there is a need for a reality check. It is a fact of life is that many people have to work within log frames. Many donors insist on log frames and they have practical value and would not be so widely used if they were useless. The important issue is how to integrate principles from outcome mapping into a log frame?

Points from the discussion

· This is perhaps more of a management tool than an evaluation tool.

· We should be modest in our expectations and in what we promise. Change is not easy.

· There is great difficulty in institutionalising monitoring and learning in an institutional culture.

· Outcome mapping tells you what you need to measure but not how you need to measure it.

· This tool would be useful in monitoring governance.

Presentation 3:

Ratna Reddy: Assessing the Impact of Participatory Development Programmes: A Sustainable Livelihoods Approach (SSL)

It is a participatory method which looks to the different perspectives/ capitals that are affecting the Livelihood’s sustainability. The capitals considered are:

· Natural capital: natural resource stock (land, water, CPRs, etc)

· Physical capital: Basic infrastructure (infrastructure, physical assets, etc)

· Financial capital: Financial resources (income, savings, credit, etc)

· Socio-political capital: horizontal and vertical social resources (networks, group membership, migration, etc)

· Human capital: skills, knowledge, health, etc

Based on the analysis of those capitals, it is possible to define impact. The analysis/ comparison of those capitals could be done by using one of the following methods: ‘double difference’ method (before-after and with-without comparisons), Statistical validity (pair‘t’ test and‘t’ tests are used) and Causality tests (regression analysis).

The method allows analyzing different dimension and its relationship; nonetheless there are some limitations. The highlighted points were:

· The quality of the baseline data is not always at the appropriate level

· The attribution problem is the major problem

· The definition of indicators and the balance between capitals is not easy

Discussion

This method is very useful for monitoring interventions for poverty reduction.

Presentation 4

Deirdre Casella: Qualitative Information Systems (QIS)

This method is based on the participatory assessment approach. It results from the challenges faced when trying to define indicators related to sustainability and equity. The objective when defining this method was to be able to come out with a demand driven approach.

The following are the most relevant QIS’ characteristics:

• It is a mixture of both; quantitative and qualitative factors
• It allows to look at several factors together in order to facilitate to describe the sustainability level of a specific intervention
• It works based on the definition of a descriptive order of score. It means that different scenarios are created in order to describe the worst and best situation related to the intervention. In the middle is the intervention’s objective, which is considered the benchmark. Thus, based on those scenarios it is possible to self assess the current situation and explain why a specific score is given.
• It allows putting in number qualitative information (scenarios), in this way it is easier to disseminate the results obtained from the evaluation.

The limitations using QIS are mainly related to the source of information used for supporting the score defined to specific intervention. The method requires a strong support for validating the results. The application of process documentation techniques is required for avoiding this limitation.

Discussion

We concluded that it is useful for getting information about outcomes that are precursors to saying something about impact on sustainability and equity. It is useful for informing policy as well as programmes.

GENERAL PANEL DISCUSSION

After the presentations some questions raised from the table. The most relevant questions were:

1. What do QIS and SL methods measure?
2. What is the cost of applying the methods presented? – Information related to costs will be bring to the table
3. Which of those methods are “really” related to Impact Evaluation?
4. How could the risk of using a specific method be handled?
5. How can we select a method?
6. What can be the role of beneficiaries for defining IE?
7. What methods can be used for defining the intervention theory of a specific programme?

The group exercise will hopefully answer some of the questions. Nonetheless those questions will used as inputs for the production of the Thematic Overview Paper on Impact Evaluation

Before opening the group exercise, there was the need to take some time for analyzing/ defining the Impact Evaluation concept. The result of this discussion was that people used this to mean different things. There probably would not be a general definition that would be universally accepted, but there is a need to be clear about the different levels and approaches that can be used. When people used this term they should be clear about what they were measuring. There did seem to be agreement that monitoring was needed both of long term outcomes and of intermediate impacts or outcomes, but there was no need to have a clear definition of where the boundaries are.

Deirdre Casella from IRC said that the aim of the round table was to expose what is known about impact evaluation and where we do not have enough shared knowledge. There is a need to match this to the work we are doing so we do not spend vast resources on measuring the wrong sort of things.

]]> http://washimpacttable.wordpress.com/2008/05/13/round-table-on-going-discussion-in-delft-summary-block-2/feed/ 1 ircuser Block 2 http://washimpacttable.wordpress.com/2008/05/13/online-discussion-question-4-what-kind-of-indicators-do-you-use-for-impact-evaluation-in-teh-context-of-institutional-change/ http://washimpacttable.wordpress.com/2008/05/13/online-discussion-question-4-what-kind-of-indicators-do-you-use-for-impact-evaluation-in-teh-context-of-institutional-change/#comments Tue, 13 May 2008 16:56:51 +0000 Harry Oosterveen http://washimpacttable.wordpress.com/?p=56 ]]>

Your answer will be discussed during the block 3 of the Round Table.
The deadline for submitting comments is May 14th 2008 at 08:30 – Central Europe Summer Time

For more information about this Online discussion on Impact Evaluation, please go to the right hand side of this weblog – General Information Box, or visit http://www.irc.nl/page/40054

The expert round table opened in Delft in brilliant sunshine and with brilliant minds (we hope). However, the Internet link crashed just as we began so this brilliance has been delayed. We hope it was worth waiting.

WELCOME

The round table was opened by Ben Lamoree, IRC Director. He cited a World Bank definition of Impact Evaluation (IE):

“Impact evaluation assesses the changes in the well-being of individuals that can be attributed to a particular intervention, such as a project, programme or policy.”

He highlighted however a number of points that made it somewhat less clear-cut:

· The way of perceiving and understanding impact evaluation changes according to your point of view.

· The what and how questions are always open and reflect the tension behind IE.

· For whom IE is done/ useful

EXPECTATIONS

The opening exercise was focused on mapping the expectations of the Round Table participants. The following are among the expectations outlined:

· Talk about reality: Theory is good, but it is important to be able to see the “real” side of it during the discussion here

· Use specific examples and specific applications: it is important for participants to be able to have an idea about how the discussions can be applied in their work

· To understand how IE can be done in the context of complex interventions

· How to use IE for learning? How to use this learning?

· To balance – Why and How of IE: The why should lead the how

· IE should be used for showing whether or not for example the local Government is ‘on Track’

· IE is a wider concept: during those two days it is important to recognize the different components of this concept

· To think outside the box, not only to think about results but also about processes

· How it is possible to get value from the money/ resources relates to IE

· To analyse how the tension between management expectations (project point of view) and Government expectations can be balanced, and especially with what professionals can deliver

· IE vs corruption: there is a need to understand how IE can lead to a lower corruptions levels

In block 4, there will be the opportunity for reconsidering those expectations and define which points should be discussed deeper.

INTERVIEWS

The objective was to bring to the table specific examples of interventions focused on Institutional Change by asking to some participants to shares their experiences related to IE in a real case. Four participants were quizzed by Joep Verhagen.

Picture by Peter McIntyre – Steve Mogere, Monitoring and Evaluation Advisor for the Japan International Cooperation Agency (JICA) in Kenya, being interviewed by Joep Verhagen of IRC

Interview 1: Stephen Mogere: JICA – Kenya

JICA is Japanese government donor agency which does not do direct project implementation but has three main components in Kenya:

1. Infrastructure

2. Capacity Building

3. Networking

When trying to apply IE, it was a challenge to handle the tension between Governmental objectives and project objectives was a big challenge; it was difficult to link the project’s results from the Government perspective and the real implication at the ground/ beneficiaries level.

Government often sees evaluation done by donors as an audit. This makes it difficult to link IE with reality. It can come to be seen as a routine process rather than something they can learn from.

Interview 2: Sompit Puyaratabandhu – Office of agricultural economics Thailand

Her example came from her 20 years experience in IE as a Governmental officer. She feels that the work she had done was often more about outcome evaluation rather than impact evaluation. Her department had to respond to the immediate needs for this kind of evaluation and she would like the opportunity to do outcome evaluation after a long time frame to discover for example whether farmers who had benefited from an irrigation project still had access to water and an improved income. Often the problems were not immediately apparent, especially if maintenance was not properly done.

Interview 3: Donal O’Leary – WIN

WIN work has three main components:

1. Awareness raising

2. Networking

3. Resources on the ground

IE has been done mainly as a requirement of the donors. The idea is to link the three programme elements in order to be able to “prove” impact in the ground. However it is was also needed by those engaged in the work. If these issues are not addressed it damages the sustainability of water supply and sanitation and ultimately the health or people. But it was expensie and time consuming. For example, if a water facility was properly reporting its performance, good audotrs talk to people on the ground (including the users) and track losses and corruption.

Interview 4: Patrick Moriarty – EMPOWERS

The objective of Empowers was to support the improvement of Integrated Water Resources Management (IWRM) by strengthening local governance. At the beginning it was not clear how to achieve this objective. Process documentation was used to track changes in the thinking and interactions of actors at many levels.

However he said he was skeptical about impact evaluation when trying to make changes in governance. Impact would perhaps be felt over a ten year period and the sector is seething with activity. This made it very difficult to separate out the impact of a particular intervention.

Regarding to IE, the following were the most important points:

· How to consider all the factors related to the programme

· How to use IE for supporting learning

· How to balance the external and internal factors which are affecting the intervention

“The objective is create self reflective learning processes, so that learning is intrinsic to the process. It has a very strong learning element. I would make a difference between internal learning and external log-frame driven evaluation.”

MAIN CHALLENGES

In four groups participants discussed some of the main challenges they needed to address

We have tried to summarise them here.

Group 1

1. The intervention needs to have a clear and coherent strategy if you are going to do an impact evaluation – your need to know the objectives and the planned a means of getting there.
2. Do you need control groups? This is difficult in situations where you are working with institutions and they are interconnected. Should you be doing it and if so how?
3. How do you know what is te the impact of your invention and that of other inputs.

Group 2

1. If you are seeking institutional change then the evidence will be found on the ground in good services and responsive organisations. It takes a long time to show that, but you do not have to do it all at once. If you have good processes indicators you can go ahead with intermediary outcomes.
2. It is important that stakeholders are engaged but then they are not likely to bring out negative aspects in an evaluation. The challenge is to change the artuide. This is hardest to achieve with government stakeholders so that they do not refard IE as fault finding but a means of leaning, so that they cone ot their offices and engage in the process.
3. It is very had to get the evidence based documentation.

Group 3

1. Understanding why the intervientions are taking place is difficult. Log frames are often done after the fact, and are no use for learning.
2. Should focus on the impact we want to have not on what we did base don an assumption that we already assumed it would have an impact.
3. What are different reasons for having evaluation: upward (donors) and downward evaluation (to beneficiaries) but also learning as a key purpose (how do we do thins better?). Depending on the audience impact evaluation may be different .
4. Best practice how do you do what you do best?
5. What is The framework ? The project does not control all of the field and it has a temporal limitation.

Group 4

1. Impact analyses are often perceived as audits that you have to do for someone else and not integrated into the pojrect. How can you change mindsets so they do it for their own benefit?
2. How can you attribute effects to the programme. Impact is difficult to measure. We could maybe measure the possibility that what we have done night lead to the outcome, especially if the programmes have a short time frame.
3. Donors fund the process and they do not want to hear bad news.
4. Unplanned outcomes (positive and negative) are not measured.

]]> http://washimpacttable.wordpress.com/2008/05/13/round-table-on-going-discussion-in-delft-summary-block-1/feed/ 0 ircuser Interview http://washimpacttable.wordpress.com/2008/05/13/online-discussion-question-3-what-methods-of-impact-evaluation-do-you-know-about/ http://washimpacttable.wordpress.com/2008/05/13/online-discussion-question-3-what-methods-of-impact-evaluation-do-you-know-about/#comments Tue, 13 May 2008 07:35:06 +0000 Harry Oosterveen http://washimpacttable.wordpress.com/?p=54 ]]>

Your answer will be discussed during the block 2 of the Round Table.
The deadline for submitting comments is May 13th 2008 at 16:30 – Central Europe Summer Time

For more information about this Online discussion on Impact Evaluation, please go to the right hand side of this weblog – General Information Box, or visit http://www.irc.nl/page/40054