In their new MCA Monitor Analysis, Does the “MCC Effect” Exist?:  Results from the 2012 MCA Stakeholder Survey, Parks and Rice, both with the College of William and Mary’s Institute for the Theory and Practice of International Relations, share the findings of a 100-country survey of more than 600 development practitioners and policymakers who are familiar with MCC and/or developing country reform efforts.  Their approach and findings are an important contribution to a relatively small body of evidence on the MCC effect (for example, here, here and here).  In short the authors find:

• MCC’s eligibility criteria are influential in encouraging policy and institutional reform in developing countries; they’re actually among the more influential of various donor incentive tools.  The incentive effect, however, is not consistent across countries or policy areas.  Countries where the rules are not well understood or where policymakers are skeptical about the criteria’s role (vs. the role of politics) in determining eligibility are less responsive to MCC’s incentive.  And while the eligibility criteria are reportedly particularly influential in policy areas like fiscal policy, business registration, and control of corruption, they are substantially less influential in areas like democratic rights.
• MCC threshold and compact programs tend to be perceived as successful more often than not.  Threshold programs are thought to successfully influence policy reforms (their intended programmatic result), and the incentive of compact eligibility is often seen as a motivating factor for threshold program success.
• Policymakers and practitioners generally welcome MCC’s use of ex-ante conditionality and selectivity, which contrasts with skepticism expressed by some researchers about this type of approach.

I’m excited to see the new information that Parks and Rice offer about where, why, how—and how well—the MCC effect really works.  However, it’s important to keep in mind (as the authors note) that the findings do not represent conclusive, definitive evidence; limitations of the sample and the perception-based nature of the data are important considerations when interpreting the results.  On the sample, the target population was essentially people with known (or likely) interactions with MCC/USG on eligibility issues, involvement in MCC programming, or other knowledge of MCC.  In many ways, these are the right people to target since they can provide an informed perspective about their experiences; however, they don’t necessarily represent the level of MCC-awareness/perception among a country’s policymakers in general.  Of those targeted, there was a response rate of about 30%.  This isn’t bad for surveys of this type, but since we can’t assume that respondents and non-respondents are fundamentally the same on average, this may be another source of bias (for instance, if those with more knowledge of/stronger feelings about MCC were more likely to respond).

There are also limits to perceptions-based data which are inherently subjective.  For instance, a survey respondent saying that MCC’s eligibility criteria provided an incentive to reform does not tell us that a reform was really undertaken, how effective it was, nor the relative weight of the MCC incentive among the surely-multiple criteria that led to the decision to reform.  Similarly, perceptions that an MCC-funded program was successful cannot substitute for the findings of an independent evaluation; in fact, while the majority of respondents familiar with Threshold programs considered them successful (albeit with no specific definition of what “successful” means), the handful of Threshold program evaluations done so far suggest more mixed results (see here, here, here and here).

That all said, the survey findings do provide insight into the viewpoints of many of the key people MCC is specifically trying to influence.  MCC can be proud that many developing country policymakers claim to like MCC’s approach and feel it helps spur policy reform.  The survey also gives MCC useful information about where the MCC effect seems strongest and where it could be stronger.  Building on those findings, the authors suggest how MCC might increase the pull of its incentive effect.  One way is to raise more awareness about MCC eligibility.  I can see some relatively simple administrative ways MCC could make its rules more accessible, for instance, by publishing its selection criteria report and associated materials in multiple languages (this information is posted online annually, but only in English).

The authors also suggest that MCC should be more transparent around the justification for eligibility (and non-eligibility) decisions.  MCC has made substantial strides this area, for example, by publishing detail on the types of supplemental information the Board of Directors takes into account and making explicit through the use of a new democracy “hard hurdle” the Board’s historically revealed preference for passing up non-democratic countries that otherwise meet the criteria.  However, MCC still provides little specific detail on why countries that perform well on the indicators are not selected.  There are, of course, a number of very valid reasons (e.g. sensitivity to bilateral relationships) why a USG agency might choose not to disclose this kind of information, which the authors acknowledge, but—according to the survey results—the tradeoff may be a somewhat less effective incentive.

To dive into the study further, the full survey report, Measuring the Policy Influence of the Millennium Challenge Corporation: A Survey-Based Approach, can be found here.

Here are the decisions:

• Liberia, Niger and Sierra Leone are eligible for MCC compacts for the first time ever.
• Tanzania and Morocco are newly eligible to begin developing second compacts.
• Benin, El Salvador, Georgia and Ghana were all reselected which allows them to continue developing compact proposals.
• Guatemala was selected as eligible for the revised MCC threshold program.

We discuss the debates around selecting Liberia, Niger and Sierra Leone for MCC eligibility in our MCA Monitor analysis. There are still some risks in each of the countries, but they’ve made significant policy reforms, pass the MCC indicators and are being rewarded in the year they show improvements. Tanzania was also expected to be reselected for a second compact because the first compact has been well-implemented, Tanzania continues to pass the indicators and the MCC leads the US Partnership for Growth constraints analysis and energy effort in the country.

In the case of Morocco, we expected there to be pressure to select it for a second compact but would not have selected it this year for a couple of reasons: it narrowly passes the indicators (fails the political rights indicator and only passes the civil liberties indicator by three points); it has a complex $698 million compact that doesn’t end until September 2013 that will take time and attention to finish well; and Morocco’s high income and relatively low poverty headcount (2.52 percent) and low number of people living under $1.25 a day (813,279) signal it should be ready to move beyond MCC and foreign assistance on to other forms of engagement such as trade and private investment. The MCC’s rationale is that the time for serious work on economic development in Morocco is now. I sympathize that there is a need for US government support for smart economic development at the right time and that MCC has some of the best thinking on that right now. But the MCC purist in me worries that the choice seems more politically driven (something the MCC is designed to avoid) rather than based on a sense that it would be the best use of scarce MCC resources.

While I’m thrilled our predictions were spot on in almost all cases, it’s even more important that we raise the right issues. The biggest issue this year was how the MCC would deal with a record number of countries that pass the indicators test and could be eligible for MCC assistance even though the budget is expected to be small (roughly $900 million if all goes well). The MCC came out in the same place we did and decided to reward countries this year that show progress and remind their partners that competition is part of the MCC selection and compact proposal process. Stay tuned to see which of the MCC’s new or continuing partners develops the best compacts for FY13 funds.

This is the 10th year CGD has produced independent data and analysis of the MCC’s selection process. This year, several countries pass the MCC’s indicators test for the first time ever and a record number of countries are contenders for second compacts. The competition is high and the resources are scarce. The board meeting will also be the last one during President Barack Obama’s first term and presumably the final session chaired by Secretary of State Hillary Clinton (though recent reports indicate she may be on medical leave this week).

The MCC board, which will see other changes among its governmental members in Obama’s second term and is still waiting for Senate action to fill the two vacant non-governmental board seats, will be looking to keep the MCC model on the leading edge of the Obama administration’s development agenda. But their decisions also take place amid a protracted US budget process, pressure to cut federal spending and an incoming Congress that boasts more than 270 new members since the MCC was authorized in 2003.

The MCA Monitor’s advice? Stick to the MCC mission by using the indicators as the primary (but not the only) guide to selecting eligible country partners. Keep looking for qualified new partners, especially low income countries that perform well on the indicators. Encourage competition not just for selection, but for funding compacts. And keep pushing the boundaries for transparency and evidence-based decision making, including when tough love is needed (hint: Indonesia).

Based on the data and these principles, we predict the MCC will select the following new countries as eligible for first compacts:  Liberia, Niger, and Sierra Leone. The MCC board will also likely select Tanzania as eligible for a second, though smaller, compact and re-select four countries already developing second compacts: Benin, Ghana, El Salvador and Georgia. It may also put Morocco into the new threshold program. Read all the details and data here.

UPDATE 12/21/12: Read about the MCC board decisions–and how they match up with our predictions–here.

A bit of background on the nominees: Lorne Craner is the president of the International Republican Institute and has already served one three-year term on the MCC board. Morton Halperin, senior advisor for the Open Society Foundations, would be a new face on the board but certainly not new to the MCC.  Halperin has been following the MCC for many years, urging the agency to make democracy an explicit “hard hurdle” in their selection process (a rule the MCC recently embraced).

The unique MCC board structure—five governmental and four non-governmental members—brings both insider and independent expertise to MCC decisions on country selection, economic growth and poverty reduction analyses, risk assessment, and more. Its non-governmental board members are some of the best messengers on Capitol Hill, not to mention with NGOs and the private sector—all important constituencies for the MCC. If Craner and Halperin are confirmed, it will be the first time since September 2009 that the MCC would have all four non-governmental board members in place.

The upcoming December 19th board meeting, however,  is expected to be the last before some big shake-ups to the governmental side of the MCC board. Secretary of State Hillary Clinton, Treasury Secretary Timothy Geithner and US Trade Representative Ron Kirk have all indicated that they will be moving on from their current government posts and therefore from the MCC board. While it seems unlikely the Senate will confirm Craner and Halperin before the board meeting next week, I hope the Senate will confirm them by the end of the year and before the next round of musical chairs.

In her remarks (which are worth reading or watching in full), Clinton recalls her promise to “elevate development and diplomacy to be on a par with defense…as part of our smart power framework for foreign policy and national security.” (This is a promise she explained in detail at CGD.) She told MCC staff she believes the US has gone a long way towards achieving that but needs to continue to reform.

Clinton gave the MCC gold stars for showcasing “some of our best thinking about how to do development for the 21st century” and providing “information and inspiration” to reform the rest of the US development apparatus. Clinton also praised MCC for putting a premium on results, data analysis and transparency. She told the MCC staff:

Because of the standards that you’ve set and the accountability and evaluations that you have imposed upon yourself, we are beginning to get a better idea of what works and what doesn’t work.

Now, that is not always popular, and it’s not always easy, because we’re all human. People get used to doing things a certain way, and then can, on an anecdotal level, see results that reinforce the patterns that they’ve engaged in. But we can no longer afford to do development like that. We have to have better data, harder analysis, more accountability, both for us, but also for the countries and people with whom we work. We look to MCC for helping to bring about that strategic shift that we’re making in our development work from aid to investment and looking at the risk-reward calculation, literally expecting to be able better to calculate a rate of return.

We have to ask hard questions. We have to be unafraid to expose our own shortcomings and the problems that we have. Some people worry about that, that that will mean that the Congress or the American public won’t support us. I actually think it’s contrary. I think greater transparency, internally and externally, gives us a stronger platform to build on for the programs that we think are worth investing in, and MCC is certainly at the top of the line there.

It has to feel pretty good for MCC staff to hear the Secretary of State not just say “good job” but to reflect a nuanced understanding of the MCC’s mission, approach and why the tough decisions that set MCC apart—from selecting countries, sometimes cutting off funding and publicly evaluating impact—matter. It also puts pressure on the MCC to keep the bar high.

Clinton even praised the MCC’s innovative board structure, saying she highly values the board’s outside, independent, private representatives. Hear, hear! Now if only she could nudge the White House to fill those two long-vacant private MCC board seats!

In response to a question about what the Obama administration’s development agenda should focus on in the next four years, Clinton offered three suggestions: more rigorous analysis to go along with the reforms; better coordination across the government and congressional committees; and more achievable cost-savings including through better foreign aid data, cost-benefit analysis and procurement reform. These sound good to me, but oh how I wish we could hear more of Clinton’s insights on how her successor and other development policymakers might overcome the logjams that have, up until now, prevented these reforms from moving faster.

The five impact evaluation studies report on the implementation of farmer training programs that represented a small share of the overall grants (about 13 percent of the total budget and 2 percent of MCC’s portfolio of compacts). They found that all five of the programs were successful in completing their activities and training farmers, largely as planned. In fact, the number of farmers using improved cultivation techniques actually exceeded targets in El Salvador (by 64%), Ghana (by 29%) and Armenia (by 17%). Farm incomes appear to have increased in El Salvador (for dairy farmers), northern Ghana and Nicaragua while no impact was found in Armenia or central Ghana. The biggest puzzle is that none of the programs demonstrated reductions in poverty (as measured by household consumption) even when farm incomes rose. The MCC largely attributes this disconnect to failures in program logic, study implementation and the difficulties of measuring rural incomes. While these caveats are important, it is at least as important to begin questioning some of the fundamental tenets behind the claim that farmer training can reduce poverty.  My guess is that these MCC studies will be actively cited for years to come, perhaps contributing to a revolution in thinking about farmer training programs comparable to the shakeups experienced by job training programs decades ago and microfinance today.

Why do I predict such a big effect on farmer training programs? Because the impeccable logic of farmer training programs – from increased productivity to increased farm income to reduced household poverty – has rarely been tested. Taken together, these five studies (combined with three earlier studies from other institutions) suggest that some farmer training programs increase farm income. They also suggest that the impact on household incomes may be negligible.

This result should not be a surprise. Farm income is influenced by many things beyond productivity. Think about the way commodity prices fluctuate; consider how increased local supplies could even drive prices down. Household consumption is influenced by even more things –tradeoffs between children working or spending time in school, allocation of time between remunerated and unremunerated work, sharing across households. In his peer review for MCC, Will Masters concluded:

“… successful program designs that target productive public investment will – like this excellent impact evaluation – be aimed at beneficiaries’ full income from multiple activities, taking account of heterogeneity and risk.  To raise full income for a whole population, program targets should be defined in terms of the market and policy failures that they remedy and the productive inputs to be supplied, with success measured in terms of final consumption from both farm and nonfarm enterprises.  The Nicaragua RBD evaluation shows clearly how programs that promote specific businesses can meet their targets, and yet fall short of their larger objective.”

It will take years to distinguish whether the limited impact on poverty is due to poorly designed interventions or to some failure in the strategy as a whole. But that discussion can move faster and more deliberately as a result of these additions to the knowledge base.

As I wrote in part one, MCC seems to be taking the right approach to these evaluations. I particularly congratulated them for their public commitment to keep doing rigorous evaluations and publish their results. In part two, I emphasized the lessons for doing better studies in the future. The next step for the substantive discussion raised by these studies is to ask whether farmer training is a “good buy” for development efforts. What standard will future farmer training proposals get when they are presented to decision making bodies at the MCC… or at USAID, the World Bank, SIDA, the Gates Foundation? Until now, in our relative ignorance, it was plausible to think that successfully implemented farmer training programs significantly reduced poverty. What these studies tell us is that the standard of proof is shifting. The fault may lie in the logic of farmer training programs or something else. But these studies are a wakeup call. We know something about the links between training and increased productivity, though we could do much better. We know very little about whether such programs can reduce poverty. If MCC and others take these lessons to heart, future programs will have to be designed very differently (… and tested!).

Impact evaluations are not foolproof. The probability that a given evaluation will successfully yield good information is nowhere near 100%.  Evaluation is a risky endeavor – as risky as a development project or even a new business. All agencies (managers take note!) should just recognize this from the start.

There are at least three basic ways that an evaluation can fail:

• If the program doesn’t get implemented, the study has nothing to evaluate.
• If the program is implemented but the study doesn’t collect good data, it can’t tell you much.
• Even with good implementation and data, the evaluation may have too little “statistical power”, that is, it may have too few observations to detect an impact when the effects are small or when there are lots of other confounding factors.

In the MCC studies, most programs were implemented, achieving the inputs and outputs that had been planned. For example, in El Salvador, 11,520 farmers reportedly applied improved farming techniques as a result of the program, well above the target of 7,000. However, some implementation problems are not apparent from the raw numbers. In Honduras, the process of selecting farmers for the program was poorly done and in Armenia, complementary irrigation investments were delayed.  You can only evaluate the impact of an intervention if it has been successfully implemented. Most of these programs accomplished the tasks – the inputs, activities, and outputs – that they set for themselves but implementation of some components did not fully meet the intended design.

The MCC studies confronted some problems with data collection but the one which stood out for me was the difficulty evaluators had in maintaining a clear enough distinction between farmers who received training and those who didn’t. In Honduras, the implementing agency apparently confused matters by obfuscating how farmers were selected. In Ghana, both participants and non-participants seem to have received about the same amount of training. Evaluators are constantly under pressure to bring everyone into a program even when no one really knows if the program is beneficial. While it would be unethical to test a proven intervention by withholding it from a control group, the results of these and many other studies show that so-called “proven” interventions are frequently less robust than proponents claim. In most of these cases, non-participants were not harmed and many participants failed to benefit. This is not like a program to feed hungry people or provide parachutes to those who jump out of airplanes. Getting farmers to spend time listening to an extension agent and change the way they farm may or may not be beneficial to them. Like an unproven medicine, it should be tested and the control group included only if some clear benefit becomes apparent.

The biggest problem these studies faced was inadequate statistical power. In some cases, the average difference in income between participants and non-participants were large but there was so much variation, so much “noise”, that the difference could have occurred just by chance (Michael Carter addresses this in his peer review). This problem is common in the impact evaluation world. Markus Goldstein notes that the randomized roll-out designs were partly to blame. Given the complexities of our social world, I think we are simply too optimistic about what our programs can accomplish. When you expect farm or household incomes to double, you don’t need very large samples to distinguish the impact. The combination of high predicted returns and cost constraints on evaluation is a perfect recipe for small samples that lead to “we-don’t-know” at the end of the day. Managers who want to increase the probability of getting good evaluation results and effectively use their evaluation budget should take heed. When teams propose to study such complex social interventions, don’t ask them to reduce the cost. If anything, ask them how much it would cost to increase the probability of detecting a statistically robust estimate and budget for that! You won’t regret the decision when it comes time to show what your department has learned.

MCC is taking the right approach to these evaluations. First and foremost it has reaffirmed its commitment to keep doing and publishing rigorous evaluations. For this alone, they deserve an international award. Secondly, MCC is presenting the results in a clear but nuanced fashion – emphasizing learning relative to accountability. Third, they are proposing to improve evaluation designs so that future studies yield better evidence. And fourth, they are mining the studies for information about ways to better design and implement their farmer training programs.

Now, what do the studies tell us about farmer training programs themselves? I’ll address that in part three …

Sarah Jane Staats argued that “It’s Not About the Grade” when she gave MCC a gold star for pursuing rigorous evaluation, being transparent and (still to be seen) applying the lessons from these studies. Now that I’ve had a chance to read the MCC’s brief and some of the studies, I agree.

Congratulations, MCC, for living up to your commitment to rigorous evaluation and transparency! This is a big achievement. The world is full of agencies with bias in their evaluation systems where the positive ones get reported; lukewarm ones get revised; and the negative ones get buried. Even when programs fail to yield the expected benefits, the knowledge you are sharing from these studies are likely to yield huge benefits by influencing the design of future programs and the allocation of future aid money. In other words, the benefits may go far beyond the impact of any one program.

So while we’re eating the party cake, we can debate at least three different questions about these studies. In this blog, I’ll talk about why I think this is a game changer for institutions that finance development projects. In later blogs I’ll talk about what we can learn from these studies about doing impact evaluations and then specifically address what the studies tell us about farmer training programs.

For development institutions, MCC has established a new standard for what it means to be a responsible public agency. They have explicitly stated how they think their programs will affect the chain of events from inputs to impacts; contracted qualified people to test their assumptions; and taken advantage of outside perspectives to get real debate over their programs. Using MCC’s actions as a model, here is my list of what a responsible agency does:

• Recognize how little we know and commit to study programs rigorously. The importance of studying an operation depends on a few things but mostly on the value of the information that can be learned from it. This means focusing research efforts on programs that are unproven and which either represent a large share of your portfolio or are newly hyped and rapidly expanding. Unlike other organizations, MCC recognizes that evidence is weak for most development programs and has plans to study about 40 percent of its portfolio. This is certainly a record for a development agency, possibly even for most other public and private organizations. By having the temerity to question whether farmer training programs really increase farm incomes and whether they reduce poverty, MCC has uncovered important flaws in project logic and useful lessons for designing better programs.

• Be transparent about what you’re studying. The simplest way to resist the temptation to hide bad results (which is where we often learn the most) is to be open about what you’re studying from the moment you begin. It is a basic standard of medical research to pre-register any clinical trials so that all results – not just the desired ones – are in the public domain. MCC has published the study designs on their website. Anyone can see the list of planned studies along with descriptions of the study design.

• Use independent researchers. Most organizations cannot afford to have a sufficiently large number of staff with specialized skills in impact evaluation. You do need to have enough staff with expertise so that they can properly design terms of references and select qualified researchers. MCC has done both. Those responsible for overseeing the impact evaluation efforts are respected in their fields and the list of research groups conducting these recent studies are also top of the line.

• Take advantage of peer review. Impact evaluation is a science. It progresses through debate, listening to challenges, and responding to critics. MCC commissioned peer reviews of these five studies which are publicly posted alongside the evaluations. I found William Masters’ comments on the Nicaraguan and Ghanaian projects particularly insightful, but learned something useful from all of them.

• Publish the underlying data and computer code. All of us make mistakes and the best way to minimize such errors is to let someone else check your work. This is one reason that publishing data and associated computer code is essential to learning from impact evaluations. CGD is one of many organizations that have adopted a transparency policy for data and computer code. While MCC has not yet posted the datasets for these evaluations, its commitment to do so is included in their Open Government Plan. MCC staff inform me that they are delayed in establishing standards regarding privacy and will post datasets once that process is complete. I hope that happens soon.

• Groups of rigorous studies give you a better chance of getting useful information. By doing five different studies on related questions, MCC has a better chance of extracting useful information from the evaluations. Some of the studies provided inconclusive evidence, others were more robust. Some of the findings suggest ways to improve implementation of programs and others question the logic behind the interventions. I think this helped MCC produce an initial summary that strikes an excellent balance. Instead of exaggerating positive results or underplaying negative findings, MCC was able to consider the robustness, validity, and generalizability of the studies as a whole.

When I first heard about MCC’s plans to undertake systematic evaluation (back in 2005), I was quite skeptical. I had many reasons to expect that political, budgetary, and bureaucratic pressures would dilute the effort. For that reason, I agree with Markus Goldstein when he describes the MCC initiative as “gutsy.” Establishing a systematic process of evaluation like this isn’t easy for a public agency but now we know it’s possible.

Let’s take a pause to celebrate over the cake and punch. But then, let’s talk about what these studies tell us about the process of doing good impact evaluations, which I’ll write about in part two …

MCC sets the transparency and evaluation standard higher than any other US development agency (USAID has a new policy to catch up). And MCC impact evaluations go beyond typical performance evaluations to test—with control groups and counterfactuals—whether their activities directly increase incomes. The MCC risks being unfairly compared to organizations that aren’t as rigorous and transparent, but is forging ahead to gather and share real evidence. That means good results (incomes up!) and bad (incomes not up, can’t attribute to MCC, or can’t measure).

The first five MCC impact evaluations cover farmer training activities in MCC compacts with Armenia, El Salvador, Ghana, Honduras and Nicaragua. Farmer training is just 13 percent of MCC investments in these five countries and 2 percent of MCC’s global compact portfolio. But the lessons will matter for the MCC and other donor programs with similar investments such as USAID’s Feed the Future initiative. And these first five MCC impact evaluations already double the stock of evidence globally on farmer training activities.

According to preliminary conversations with MCCers, the impact evaluations:

The Biggest Experiment in Evaluation Blog Series

• MCC and Systematic Learning
• MCC and Lessons for Better Impact Evaluations
• MCC and Lessons for Farmer Training
• Show the MCC met or exceeded projected farmer training outputs and outcomes (e.g. number of farmers trained and increased crop yields);
• Detect increases in farm income in some cases;
• Do not yet detect statistically significant increases in household income as a direct result of MCC-funded farmer training activities; and
• Two evaluations themselves failed (i.e. evaluators could not measure elements required to make judgments).

I share the MCC’s view that impact evaluations aren’t about pass or fail for specific projects. Development programs—and the MCC compacts—comprise multiple complex activities. Some may turn out well, others may flop, and the measure of a strong organization is that it wants to know the difference and learns and improves when it finds out.

As such, the MCC and its friends shouldn’t disproportionately emphasize specific success nor should critics focus solely on investments that don’t meet projected targets. But I’m not willing to say the grades don’t matter at all. It’s a big deal that the MCC is trying to gather rigorous data—and get some grades—for its investments. It’s just that how the MCC uses what it learns from the evidence matters more than any individual program result. (A recent Engineers without Borders Failures Report and the World Bank’s Failfairs push this point.) The MCC is already talking to the Hill, NGOs, and US development policy makers about the findings, which is a very good first step.

I’ll also be looking for how the MCC uses this new evidence to change future decision-making. How will the MCC balance pressure to keep moving on program implementation and stick to a sequence required to measure impact and learn something at the end of the program? Were initial estimates of MCC impacts too optimistic and if so, why?  Should the size or scope of future programs change? Will future evaluations be designed to measure impact 5 or even10 years after the compact ends? And will the MCC keep its commitment to conduct and share independent, rigorous evaluation across the rest of its programs, including in countries like Madagascar and Mali where political coups forced the MCC to halt compacts but something can still be learned from what was invested?

The United States should always aim to get the biggest bang for its development buck, but to date there has been little if any rigorous data on what works and what doesn’t. The MCC gets a gold star for its courage to conduct and share the first real evidence—and the grades—for its development investments so everyone can do better. We should hope, and keep pressing, the MCC and other US development agencies for more of this kind of good bad news to inform smart development policymaking, especially in the tough budget cycles ahead.

*CORRECTION: My colleagues have pointed me to two recent USAID evaluations—one on social insurance in Nicaragua and one on Zambia’s production, finance and improved technology project—that would qualify as independent impact evaluations, plus three others that use rigorous qualitative methods (if not full control groups and counterfactuals). All are up on USAID’s evaluation website. I was also reminded that USAID did some independent impact evaluations in the 1970s and 1980s.  My apologies for the oversight and kudos to USAID for having a few recent impact evaluations already under their belt and committing to produce many more. It seems to me there is probably a good story here about USAID’s history with impact evaluations including why, if they were doing them 30 years ago, there were so few in recent years. Would welcome thoughts and comments from readers!