If you read this blog through RSS you shouldn’t notice any changes.

Please let us know if you have any problems.

The most obvious answer is provide the type of evidence that the rest of our top-rated charities have to demonstrate your impact. But for many organizations, providing such evidence is not possible. Many may not have the resources to conduct the evaluation needed to demonstrate success. (This strikes some as “unfair,” but our goal isn’t to be fair - it’s to find charities that “casual donors” can be truly confident in.)

There is another way to improve your rating, however: demonstrating failure. (GiveWell tries to meet this standard for ourselves on our shortcomings page.)

We give Doctors Without Borders a 1-star rating because of its publication of a report demonstrating that a program they ran failed. We write:

In our experience, charities are very rarely willing to share evidence of disappointing impact. We believe that any charity that does so is being unusually honest about the challenges of international aid, and unusually accountable to donors. We expect that charities capable of spotting, documenting and sharing disappointing results are better positioned to improve our time.

If your organization is listed on the GiveWell site and you want to improve your ranking, publish a case study of a program you ran that failed. As usual, we’re not looking for marketing materials, and we won’t accept “weaknesses that are really strengths” (or reports that blame failure entirely on insufficient funding/support from others). But if you share open, honest, unadulterated evidence of failure, you’ll join a select group of organizations that have a GiveWell star.

• Basic outline and questions we asked
• Self-Evaluation: GiveWell as a donor resource
• Self-Evaluation: GiveWell as a project
• Plan: top-level priorities
• Plan: specifics of research
• High-level summary in Excel format

The documents we’ve referred to in the above posts are linked from the plans and self-reviews page of the GiveWell website.

In a previous post, we laid out our reasons for focusing on the broad category of “research” over the next year. This includes

• Research vetting: subjecting our existing research to strong, critical scrutiny from people with substantial relevant experience and credentials.
• Research on new causes such as U.S. equality of opportunity, disease research funding, and environmental issues (particularly global warming mitigation).
• Research maintenance and systemization: formalizing our process to the point where it can be maintained with as little input as possible needed from the co-founders.

After considering the time and resources available to us, we’ve concluded the following:

• Research vetting will likely involve substantial “down time” (i.e., waiting for responses from the people we’re hoping to get vetting from). Therefore, it will be a major priority of ours but will not account for a huge amount of our time.
• Re: research maintenance and systemization. Over the next year, we will be pursuing one temporary expansion in capacity via a 5-month contractor (a person who has done substantial volunteer work for us in the past, and whom we feel can add greatly to our research). During that time, there will be four people putting substantial time into research: the two co-founders, our one Research Analyst, and the contractor. It would not make sense to plan more hires at this point, given the relatively small amount of well-defined work we have available and the uncertain time costs of working with both our Research Analyst and our contractor. (More reasoning at our previous post, under the heading of “Is GiveWell’s research process ‘robust’?”)
• We will be putting significant time - essentially, the time we have left over after dealing with research vetting, research maintenance and systemization, and “low-hanging fruit” in the areas of packaging and marketing - into research on new causes.

The top contenders for new causes are as follows, listed in the order in which we are currently prioritizing them:

• More sub-causes within international aid - including disaster relief/recovery, charities aiming to help orphans and vulnerable children, and more attempts to find a strong microfinance option. Because we find international aid to be a strong area for a donor, and because we have a strong sense of the major issues within it, this cause is a high priority. We expect much of the work in this area to be done by non-co-founders (i.e., our Research Analyst and/or contractor).
• U.S. equality of opportunity (early childhood care; education; employment assistance). We have already done substantial work in this area and have a concrete idea of how to create a high-quality report. We expect to be able to outsource much of the work on this report to non-co-founders. Therefore, although we don’t consider this the most worthwhile cause, the “return on investment” for us is very high in terms of what we can produce with a limited amount of work from the co-founders.
• Disease research funding. Our intuitions suggest that this is a very promising area for a donor, but we have not done any work on it to date, and as a result our research on it will be somewhat like our research on international aid was at first: open-ended, slow, and heavily dependent on co-founders, with the rate and the nature of progress difficult to predict. We hope eventually to understand enough about this area that we can apply a consistent methodology to disease research charities, as we do with our current international aid report.
• Environmental issues, particularly global warming mitigation. As with disease research, we have little background and expect a slow, open-ended process at first.

We have assembled a work plan with time estimates. Over the next year, we expect to complete research on the first two causes listed and to make some progress (though how much is difficult to predict) on disease research funding.

In previous posts, we discussed the progress we’ve made, where we stand, and how we can improve in core areas. This post focuses on the latter, and lays out our top-level strategic choice for the next year.

Broadly, we see the key aspects of GiveWell - the areas in which we can improve - as

Research, i.e., creating and maintaining useful information for impact-focused donors. This includes

• Research vetting: subjecting our existing research to strong, critical scrutiny from people with substantial relevant experience and credentials.
• Research on new causes such as U.S. equality of opportunity, disease research funding, and environmental issues (particularly global warming mitigation).
• Research maintenance and systemization: formalizing our process to the point where it can be maintained with as little input as possible needed from the co-founders.

Packaging, i.e., presenting our research in a way that is likely to be persuasive and impactful. This includes

• Consolidating/writing up/improving the case for our research’s credibility.
• Improving our website so that it is clearer and easier to use.
• Finding new ways to express the output of our research in ways that are more emotionally/intellectually compelling.

Marketing, i.e., increasing the number of potential “customers” we reach. This includes.

• Improving the persuasiveness and clarity of our research.
• Pursuing partnerships with other donor/consumer resources.
• Pursuing partnerships with donor-advised funds and wealth advisors, which might connect us to wealthy individuals seeking help with their giving decisions.
• Pursuing partnerships with people and organizations that focus on fundraising for particular causes, rather than particular organizations. GiveWell’s research could help such organizations fundraise for outstanding organizations within their causes (and outsource responsibility for finding such organizations and justifying their choices).
• Pursuing partnerships with corporate giving programs.
• Advertising.
• Concerted efforts at “earned media,” i.e., focusing our research on issues that are likely to interest the media (which are frequently the opposite of the issues we find most interesting and important) and engaging in concerted public relations efforts around these issues.

Research vs. packaging/marketing

One of the core debates that comes up repeatedly among GiveWell stakeholders is whether we should focus on research or packaging marketing.

In brief, the “focus on packaging/marketing” view is that:

• Our research is already more useful, for impact-focused donors, than other available resources.
• Our reach, and impact, leave much to be desired.
• Therefore, we should focus on reaching more people. Doing so will lead to quicker learning about the odds of GiveWell’s ultimately succeeding, as well as quicker learning about what our potential “customers” want from our research.

The “focus on research” view is that:

• We have substantial room for improvement in the credibility of our research (who endorses it), the breadth of our research (how many causes we’ve covered), and the robustness of our research (how dependent our process is on the co-founders).
• Improving these areas could be very important to later packaging/marketing efforts. (Having knowledge of more causes could substantially affect our strategy for “packaging” our research in ways that people are likely to find interesting. Both the number of causes and the overall credibility could be important to many of the potential partners mentioned under our marketing options.)
• Many of the most promising marketing strategies can be expected to benefit from the simple passage of time. We are interested in partnering with organizations that take a long time to make decisions and prefer that their partners have significant track records / be relatively established.
• While our research, as well as the passage of time, could substantially affect the way we package/market, the results of packaging/marketing (and the passage of time) are unlikely to have much effect on how we do our research. It therefore makes sense to work more on research before focusing on packaging/marketing.
• We believe that we are only ~2 years away from having broad, robust, credible research that could then be maintained at a relatively low cost (both in terms of funds and human resources). Therefore, we think it is practical and desirable to reach this point before focusing on packaging/marketing.
• Research is our core competency. The content we created for impact-focused donors is what makes us unique and it’s what we feel we do best. By contrast, we don’t consider ourselves particularly good at packaging or marketing. By focusing on research and making all of our output public and free, we enable others (who may be much better at packaging/marketing) to use and adapt it. (Peter Singer is an example of someone who has marketed our research better than we could.)

We have had many internal discussions on these issues. At this point, all members of GiveWell’s Board and staff feel that the “focus on research” view is stronger.

Therefore, we intend to focus on research over the next year, while allocating some time to the “lowest-hanging fruit” within packaging/marketing.

A future post will discuss the specifics of how we’re prioritizing the different aspects of “research.”

This post answers the “additional questions for stakeholders” that we posed to ourselves in January about the state of GiveWell. For each question, we discuss

• Our progress over the last year (specifically, since our last business plan in 11/2008);
• Where we stand (compared to where we eventually hope to be);
• What we can do to improve from here.

Is GiveWell’s research process “robust,” i.e., can it be continued and maintained without relying on the co-Founders?

Progress since 11/08

As of 11/08, we had only made one extended attempt to hire a third employee, and had ended the relationship because our research process was not developed enough to make it possible for us to provide the necessary oversight (more).

In early 2009, we made several more unsuccessful attempts to integrate a new hire into our research process, before again concluding that our process wasn’t developed enough. We tried again with more success in mid-2009, hiring:

• One person who stayed with us from May through December 2009, focusing on our grant application process for economic empowerment, before leaving for a job in consulting.
• Natalie Stone, who has now been with us since July 2009 and is charged with maintaining and expanding our international aid report, by reviewing charities along the criteria we have defined.

Details at our work plan review.

We feel that a key factor in being able to productively hire people has been the progress on our international aid report. We now have many available examples of charity reviews and a consistent set of questions we ask and criteria we use. We have thus been able to be more concrete with new hires about their responsibilities and the expectations on them.

Where we stand

We feel that for the cause of international aid specifically, we are approaching the point where another person can maintain and expand the report with little oversight from the co-Founders. However,

• This applies only to our international aid report. Covering more causes will require heavy involvement from the co-Founders.
• The role of maintaining the international aid report is still not well-defined enough that we feel it can be passed on from one person with a given set of credentials and qualifications to another, reliably and systematically. At this point we would stick to saying that Natalie Stone can maintain it with relatively little oversight.

In short, our research process is not at all robust.

What we can do to improve

Our experience to date has been that hiring and training people (a) involves substantial time investment by the co-Founders; (b) can easily fail to add value to the organization; (c) is more likely to succeed when we are able to be concrete rather than open-ended about employees’ responsibilities, which in turn requires that we have a well-defined research process (i.e., many available examples of charity reviews; a consistent set of questions we ask and criteria we use). With that in mind:

• We can research more causes. We expect research on new causes to be initially open-ended and improvisational (as past research has been), but we also expect to gradually produce more consistent criteria, rules of thumb, and examples, with which we can train others to do the research.
• We can continue hiring people and training them by slotting them into our research process, where we have well-defined enough responsibilities to make this work.
• We can attempt to make our process more formal, to the point where much of what people need to know is captured in writing and as little as possible relies on specific people. We feel at this point that work in this area would be premature. As we work to train new hires, we learn more about how to communicate and formalize our process.

Does GiveWell present its research in a way that is likely to be persuasive and impactful (i.e., is GiveWell succeeding at “packaging” its research)?

Progress since 11/08

A year ago, it was extremely difficult for donors even to understand our work, much less engage with it. We have put significant effort into improving the clarity, organization and credibility of our research, and our efforts/progress on this front are discussed in our previous post under the heading of “Is it practical for donors to evaluate and use GiveWell’s research in the areas it has covered?”

We have not put effort into making our research more emotionally appealing, psychologically persuasive, etc.

Where we stand

Our current content aims for clarity over persuasiveness. We feel that our clarity is now reasonably strong (as discussed in our previous post under the heading of “Is it practical for donors to evaluate and use GiveWell’s research in the areas it has covered?”) But our presentation remains dry, making next to no attempts at emotional engagement.

What we can do to improve

• We can make a concerted effort to “repackage” our existing content, presenting the same ideas in more emotionally compelling ways. Such an effort would involve thinking and working in very different ways from what we’ve spent our time on to date, and may require substantial involvement from people who are more focused on, experienced with and skilled in communications than anyone on our current staff.
• A lower-cost, “low-hanging fruit” version of the above option is to put more effort into integrating our main website with our blog - we have created a lot of content on the blog, much of which is not duplicated in any way on the main website and is interesting/engaging in ways that the main website is not.
• We can obtain and collect endorsements from people whose support donors are likely to find meaningful. Doing so would overlap heavily with the activities described in our previous post under the heading of “Is it practical for donors to evaluate and use GiveWell’s research in the areas it has covered?”

Does GiveWell reach a lot of potential customers (i.e., is GiveWell succeeding at “marketing” its research)?

Progress since 11/08

Our metrics file shows that traffic to our site has grown significantly over the past year. Several factors have likely contributed to the improvement, and it is difficult (and in our view not necessary) to break down the improvement between them:

• Peter Singer’s featuring GiveWell in The Life You Can Save and continually promoting it
• Substantial revamping of our website
• Intensified posting to our blog during giving season
• The simple value of having been visible for another year

Where we stand

Our website traffic is still a fraction of the traffic commanded by two other donor resources, Charity Navigator and GuideStar, and our research is not used in any significant “partnerships.” Our impact remains small.

What we can do to improve

There are many possible strategies for expanding our audience.

• We can focus on improving the persuasiveness and clarity of our research, as discussed in the previous section.
• We can pursue partnerships with other donor/consumer resources.
• We can pursue partnerships with donor-advised funds and wealth advisors, which might connect us to wealthy individuals seeking help with their giving decisions.
• We can pursue partnerships with people and organizations that focus on fundraising for particular causes, rather than particular organizations. GiveWell’s research could help such organizations fundraise for outstanding organizations within their causes (and outsource responsibility for finding such organizations and justifying their choices).
• We can pursue partnerships with corporate giving programs.
• We can advertise directly.
• We can make concerted efforts at “earned media,” i.e., focus our research on issues that are likely to interest the media (which are frequently the opposite of the issues we find most interesting and important) and engage in concerted public relations efforts around these issues.

Is GiveWell a healthy organization with an active Board, staff in appropriate roles, appropriate policies and procedures, etc.?

Progress since 11/08

As of 11/08, we were operating without an official Executive Director, reflecting the fact that we felt current staff did not have adequate resources in place for professional development and supervision (see our 9/08 meeting).

Our Board of Directors consisted of only five people: Bob Elliott (then the Chair, but unable due to his schedule to remain engaged in the project, which ultimately led him to resign his position in July 2009); Greg Jensen (then the Treasurer), Holden Karnofsky (Secretary), Lindy Miller and Tim Ogden.

Since then,

• We have substantially added to and changed the composition of the Board (see our current Board members and positions). The Board now has seven members, each of whom is highly accessible and engaged with the project.
• We have replaced biweekly email reports with a monthly conference call, substantially increasing engagement from the Board. Turnout at conference calls has been high.
• We have substantially increased the ongoing professional development and supervision of staff, leading the Board to confer the co-Executive Director role on the two co-Founders. (See our 7/09 meeting.)
• We have adopted key policies.

Where we stand

Unlike a year ago, we feel that our Board of Directors and our policies are appropriate for this stage of our project. We voluntarily comply with all BBB Wise Giving Alliance standards.

What we can do to improve

We are always open to acquiring more Board members and mentors with substantial experience and ability to contribute to our project.

What is GiveWell’s overall impact, particularly in terms of donations influenced? Does it justify the expense of running GiveWell?

Progress since 11/08

As discussed above, we have seen substantial improvement in our “donations influenced” figures. A couple of questions remain as to the ultimate value/impact of influencing donations:

• Where would the donors we’re influencing be giving, if not for GiveWell?
• How does the impact-per-dollar of GiveWell’s recommended charities compare to that of the charities our “customers” would otherwise be giving to?

We have only hunches regarding the first question, and at some point should do a formal survey of our donors (including factual questions about their past giving as well as hypotheticals about where they would give without our existence) to get more meaningful information.

On the second question, we have done our best to answer “What do you get for your dollar?” for our top charities - however, answering this question meaningfully for our non-recommended charities is much more difficult (non-recommended charities are generally not recommended because so little information is available about what they’re accomplishing). We have argued that differences in charities can be enormous.

It is also worth addressing our impact on general perceptions regarding, and public discussion of, giving.

• We have become members of the Alliance for Effective Social Investing as well as the GuideStar Exchange Advisory Board; both reflect the fact that other notable organizations in the nonprofit sector take our message and/or research seriously.
• Since our joint press release criticizing the overuse of the “overhead ratio” metric, discussions of giving in the media have laid noticeably less stress on the “overhead ratio” as the primary metric of interest, and more stress on alternative approaches. Many of these discussions have directly included GiveWell.
• On one hand, we were one of the issuers of this press release and we have been stressing its message for a long time. On the other hand, there are many other parties behind this message and this shift, and we feel that relatively little of the credit can fairly be attributed to GiveWell.

Where we stand

GiveWell is not having nearly the impact that we would need to see in order to consider it a success. We still consider our project an experiment at this point.

What we can do to improve

The key paths to increasing GiveWell’s impact (as we see them) are laid out in the earlier sections of this post. Working on any of our many possible areas for improvement could lead to increased impact - the challenge is prioritizing. We’ll discuss broad priorities in a future post.