Two weeks ago I attended a convening hosted by Robert Wood Johnson Foundation that discussed this among other topics related to social media measurement. The goal was to start a conversation between people engaged in grantmaking, communication, social media, and evaluation about how to measure the impact of social media strategies.

As an evaluator in the social media strategies space, I had the privilege of sitting on the opening panel with social media guru, Beth Kanter, and one of our strategic learning and evaluation clients, Mayur Patel of Knight Foundation. Beth has written an excellent account of the meeting in a recent blog post.

One of the big questions at the conference was how to link social media strategies to the end-game impact that foundations and nonprofits are hoping to achieve. For example, how do you know whether getting people to fill out online petitions is influencing policymakers’ attitudes on an issue? In order to link social media strategies to social change, you need to look at the intersection between online action and offline impact.

Ultimately, the change you hope to see is real world change – not virtual. You probably are not just interested in knowing that more people are making positive comments on Facebook. Rather, you want to know that the positive comments are a) from those you want to be reaching and engaging, and b) that their attitudes have changed, or they are more likely to take a desired action, or they will use your product or service.

There are multiple ways to gather information at the intersection between what’s happening online and offline. A few of the most common are:

• Surveys (post-event or audience)
• Interviews with target audiences
• Content or thematic analysis (website or social media)
• Registered user surveys

There are many great resources on this topic, including FSG’s IMPACT guide on evaluating community information (i.e., online and digital media) projects. You can also check out more resources on RWJF’s Advancing Social Media Measurement website and on Twitter #sm_re.

You can view and download my full presentation from the RWJF convening on SlideShare.

So, what does the big data revolution mean for the field of evaluation? A recent discussion on the American Evaluation Association’s LinkedIn page asks if evaluation may soon be facing an existential crisis. In other words, will the availability of large amounts of data and the analytic engines that crunch the data make traditional evaluation designs and methods obsolete? Who needs sampling techniques when you can pretty much capture characteristics of whole populations? Why convene a focus group when you can just analyze twitter feeds? Or, why hire an evaluator at all, when a well-structured algorithm can draw the same conclusions? After all, we live in an age when machines are grading essay papers.

We at FSG have set out to explore the implications of big data in evaluation as part of a project that we are calling “Next Generation Evaluation”. Rather than the doomsday scenario described above, we are taking a “both/and” approach to the intersection of big data and evaluation. We believe that big data is a powerful force that can be harnessed for good, and that utilizing a mix of big and traditional (small) data in complementary ways would be better than using either of them in isolation. In our work, we’ve encountered several exciting examples of how social sector stakeholders are leveraging big data to better understand social problems, and “what if” scenarios of how big data can be better leveraged in the social sector. However, we’ve encountered relatively few examples of how social sector stakeholders are using big data for the specific purpose of evaluating the effectiveness of programs. Here’s where we need your help. If you have an example of how social sector stakeholders are using big data to evaluate their programs, please respond using the comment box below, or email us directly here. We would love to hear from you.

This week we continue the conversation started with you during FSG’s webinar on “Evaluating Social Innovation.” See our post from last week on how team structure and dynamics can be different in the practice of developmental evaluation.

Several questions posed during the webinar asked us to further describe what the products of developmental evaluation look like. Here John Cawley and Hallie Preskill share their experiences.

FSG’s webinar on the article “Evaluating Social Innovation” featuring Hallie Preskill, John Cawley, Meg Long, and moderated by Tanya Beer of the Center for Evaluation Innovation attracted over 300 attendees who wanted to learn more about evaluating innovations using a developmental evaluation (DE) approach. More than 75 questions were submitted by webinar participants, many more than could be addressed during the call. So, we would like to continue the conversation in a blog series on the practice of DE.

A number of questions asked us to further describe what the DE experience is really like. You asked “What did the evaluation team look like?”, “How are grantees involved?” and “How does the funder work with the evaluation team? Last week, we posted a blog post on the importance of listening in the practice of DE. This week, Meg and John share their experiences to answer your questions.

In our striving to have an impact and support others in achieving their missions, many of us can end up talking a lot more than listening. We learn to make judgments quickly, and then confidently propose our recommendations. As strategists and evaluators, we put together frameworks and five-point plans, and we aggregate myriad data into a mystifyingly simple strategy, theory of change, or set of findings.

Yet, never has the importance of listening become more apparent to me than in my work as a developmental evaluator. When I engage in developmental evaluation, most of my effort is in actively and intentionally finding opportunities to listen, as well as to cultivate an environment where people listen to each other.

And when done well, listening is much more than the words on a page of an interview transcript. I’ll liken that to listening to music out of your laptop speakers; you can do it, but it will sound a little fuzzy. Rather, intentional listening is like surround sound with a strong bass; you not only notice what is being said, but also how and why. You can feel the beat of the drum.

Recently, I read the article Art of the Nudge: Five practices for developmental evaluators from the Center for Evaluation Innovation. While none of the five practices explicitly mention listening, all of them require evaluators to have a keen ability to listen and hear what is being said about the program and its ever-changing context.

At the heart of practice #1 – servant leadership – is the ability to listen “deeply and actively.” This is the one practice where listening is called out as an “indispensable skill.” In employing this skill, developmental evaluators took time to “pause” so that individuals and groups could reflect on and have the time and space to truly consider new opportunities, alternate ways of thinking and doing, and internalize concerns and challenges that were arising. The authors emphasize, “…giving answers runs the risk of causing program actors to disengage” and instead advocate that developmental evaluators use “carefully crafted questions that encourage transformative group reflections.”

In other words, listen first, speak later.

Listening surely has a role to play in the other practices, as well:

• Sensing program energy – This practice is all about picking up on signals; be it a skeptical glance, crossed arms, an upbeat tone, or nervous laughter. Active and intentional listening (and observing) is critical to knowing whether to speed up, slow down, or stop.
• Supporting common spaces – Listening helps an evaluator identify common spaces (i.e., “the physical places, moments in time, and virtual spaces where key actors [interact]”) and listening takes place in common spaces.
• Untying knots iteratively – Challenges arise in every developmental evaluation – it’s in the nature of working in a complex, dynamic environment. In order to be effective in “untying” these inevitable knots, evaluators must listen to the concerns, needs, and challenges being surfaced from multiple perspectives, otherwise they run the risk of generating mistrust or being perceived as one-sided.
• Paying attention to structure – Identifying misalignment between what is expected to be taking place and what actually is taking place is a critical role for the developmental evaluator. Listening to what is being said about and done within programmatic structures (e.g., steering committees, coalitions) can alert evaluators to the strengths and gaps that otherwise can often be taken for granted. As described in the Art of the Nudge, such listening may shift a conversation from how well the steering committee is working together to does this program even need a steering committee?

As a developmental evaluator, I must be able to listen and intently hear my client as they share information about what is new and the directions that they are considering taking. Even the slightest hesitation may provide an opportunity for a follow-up question that proves to be invaluable for spurring learning. By listening intently, developmental evaluators are able to identify themes and patterns and weave them together in a way that helps people better understand the “whole” of an effort, rather than just their individual parts.

I encourage all of us to reflect on the listening role of the evaluator. How can we intentionally embed listening into all of our evaluation activities?

• A person’s sexuality is not the only thing affecting their life. The LGBTQ community replicates the patterns of sexism, racism, and classism prevalent in our society. Problems affecting women and other underrepresented groups are also affecting members of the community. Being queer creates a “multiplier effect,” making it even more challenging for queers to overcome social barriers, stay healthy, get an education, make a decent wage, etc. A clear example of this “multiplier effect” is in the study All Children Matter: How Legal and Social Inequalities Hurt LGBT Families. The document shows how children across races are more likely to live in poverty if they live with a same sex couple compared to those living in different sex couples (see graph from the report below). As evaluators it is important to account for this “multiplier effect” and be open and prepared to discuss race, sexism, class, and other social issues when engaging with LGBTQ folks.   

   

• Account for a diversity of voices in your evaluation; tapping only into the most visible LGBTQ members might not give you the diversity needed. Since the LGBTQ movement often reproduces patterns of racial and gender separation prevalent in our society, most intellectual and political circles in the community remain predominantly cis-gender, male, and white. As you determine who to include in your evaluation look for a representative set of members of the LGBTQ community to provide a full picture of the issues affecting the recipients of the programs you are evaluating.
• Don’t take for granted that you understand the political context for LGBTQ rights just by reading the headlines. Marriage equality is important, but there is a wide range of challenges affecting the community popular narrative is not focusing on: discrimination against transsexuals, violence against queers living in rural areas, and inadequate access to resources for queers with special needs, to name just a few of the issues evaluators should consider as they work with the LGBTQ community.

• A Fragile Union - article on gay politics
• Allan Bérubé’s work

Why does this matter? Because making positive change in communities requires the free flow of quality news and information. If the news and information environment is in trouble, so is civic life.  Foundations can only help improve education, public safety, the environment or anything else if people understand and are engaged in the issues. Through the Knight Community Information Challenge, more than 80 foundations have stepped up to invest in everything from local and state reporting to citizen dialogue and digital literacy, all to help their communities thrive. These projects have successes to show for it too - resulting in new funding for early childhood education, more environmental conservation and increased digital literacy among teens and seniors.

This new study - published today in partnership with FSG and Network Impact - provides a behind-the-scenes look at four foundations. We asked: Why are they working in media? How does that connect to their overall goals? How did they go about doing it? Has it mattered?

The four funders are:

• Community Foundation of New Jersey, supporting quality journalism through an online news site that promotes a dialogue on state policy;
• Hawaii Community Foundation, creating the first statewide student news network;
• Incourage Community Foundation, strengthening the civic health of a rural Wisconsin town through information;
• Community Foundation of Greater Dubuque, using information to engage residents in conserving the environment.

You can find an in-depth look at each project in today’s report. Here are a few insights and examples of foundation practices that stood out for us:

Lessons on Design and Planning

1) Assess your community’s needs and information behaviors – first.

In south Wood County Wisconsin, the Incourage Community Foundation set aside its initial plans to create an online news site after finding that more than a third of local, low-income families did not use the Internet. In response, the foundation shifted its efforts to include facilitating community discussions and building digital literacy skills.

2) Focus on a specific audience.

Identifying the needs of a well-defined audience helped local news projects find a niche. For example, after conducting a pre-launch market analysis the Community Foundation of New Jersey-supported NJ Spotlight decided to target state policymakers rather than trying to reach a general news audiences.

3) Identify ways to build stronger information channels within existing programs.

Community foundations often improved existing programs by focusing on the information needs of residents. The Community Foundation of Greater Dubuque, for example, recognized that the city-backed Sustainable Dubuque Initiative had support from policymakers, businesses and civic leaders, but that residents needed more information about eco-friendly habits to undertake them.

Lessons on Community Engagement

1) Make community data personal.

People are much more likely to act on information that helps them understand their own behaviors and how they compare with others. In partnership with IBM’s Smarter Cities program, Dubuque’s initiative created a ‘smart meter’ that allowed residents to view data about their water and energy consumption, and compare it to their neighbors. The feature proved addictive, and led to house by house changes.

2) Promote community dialogue based on good information.

Projects achieved their broader community goals by providing ways for people to discuss and debate issues. Case in point: NJ Spotlight coupled its reporting on the controversial topic of teacher tenure reform with moderated roundtables that gave legislators, schools, teacher unions and parents the chance to discuss what good teacher evaluations looked like.

3) Use digital training and mobile phones to reach people with limited Internet access.

The Incourage Community Foundation, for example, experimented with SMS text alerts and telephone hotlines to distribute news about job training services to under- and unemployed workers with limited computer skills.

Lessons on Project Development

1) Create well-branded partnerships to accelerate participation.

Local information projects built credibility and broader community involvement by creating partnerships with top-tier media organization. The Hawaii Community Foundation-backed youth news network, HIKI NO, benefited from PBS Hawaii’s statewide reach and prestige, which helped motivate students to sign up and remain involved.

2) Draw on available media and technology expertise.

The Incourage Community Foundation was able to create strong relationships with the MIT Center for Civic Media and IT specialists at local schools and universities, that helped guide its media and technology funding.

3) Look beyond philanthropy to help sustain your effort.

The NJ Spotlight now gets a third of its income from events, advertising and content sharing agreements. And in the case of Dubuque2.0, the community foundation found a long-term home for the project by merging it with the city-backed Sustainable Dubuque initiative, where it will receive full time staff support for the next two years.

We hope you will dig deeper into these case studies to find more lessons that can help shape the way you might approach meeting local information needs. You can learn more too, by tuning into the live webstream of our Media Learning Seminar, which focuses on these issues, on Monday and Tuesday of next week.

Related: “Tune into Knight’s Media Learning Seminar via Livestream, Twitter” on KnightBlog

The field of evaluation is no stranger to metaphors. Most often, evaluation draws its metaphors from the medical research field. We talk about “trials” using “control” and “treatment” groups that received “dosages” of the intervention and exhibit a certain “response” (or not) to that dosage. In her recent article in the Stanford Social Innovation Review, Lisbeth Schorr traces the roots of Randomized Controlled Trials to the a 1948 paper on Streptomycin treatment of tuberculosis.

At other times, we use metaphors from the natural sciences, particularly physics. Ever since Auguste Comte and his colleagues enshrined the doctrine of positivism in the early 19th century, we have tended to hold the natural sciences to the same exacting standards as the physical sciences, leading some to coin the term “physics envy.” Just as we do in the physical sciences, we test “hypotheses” and seek to create “models,” “theories,” and “empirical” inferences based on evidence.

However, the more experience I gain in evaluation, the more I’m drawn to metaphors from another field… the legal field!

Here’s why. Evaluators are often in the unenviable position of piecing together available evidence and making the best of what they can cobble together from research and experience in order to make meaning. Not unlike the legal field where a combination of evidence combined with probable cause supported by eyewitness testimony and possible situational cues lays the groundwork for building a case. None of these by themselves is 100% credible (exhibit A: the films Roshomon and Twelve Angry Men) but taken together they help build a fairly solid case, either for the prosecution or for the defense.

I shared this insight at a recent panel discussion and was met with almost raucous audience approval. As you might imagine, I was feeling pretty self-satisfied at having come up with a clever new metaphor for evaluation. The feeling was short-lived, however, as a conversation with a colleague revealed that the evaluation field had indeed dabbled with the legal metaphor in the 1970s and early 1980s. An “adversary model” of evaluation (sometimes also called the “judicial model”) was in vogue, where two different evaluators present conflicting positions using data and the whole process is set up like a jury trial. Alas, I thought, could this be one more example of the time-worn phrase “there is nothing new under the sun?”

Upon further investigation, I learned that the model was applied to some educational evaluations, but had several limitations and eventually fell out of favor. In a paper from 1980, Worthen and Rogers describe how terms such as “statement of charges” and “guilty or not guilty” were problematic and how the model reinforced a tendency to view evaluation as something you do when a problem or crisis occurs.

However, it is important to separate the failing of a specific “model” from the usefulness of a broader metaphor. I’m hopeful that the legal metaphor - a kindler, gentler version of it - will still gain popularity in 2013. Given the increasing complexity of the social sector, traditional evaluation approaches grounded in past metaphors are becoming less and less useful, and new metaphors are needed for dealing with ambiguity and uncertainty. The legal metaphor is far from perfect, but the legal field does have a long history of working with less-than-ideal circumstances; after all, there is no such thing as a randomized controlled “murder” trial.

I rest my, ahem, case.

What other metaphors are you drawn to?

The post goes on to describe the type of evaluation that is likely best suited for these novel interventions to reduce poverty and improve student success: Developmental evaluation. Even though both programs used randomized controlled trials to assess their impact, the authors caution that “just following the headline finding [based on the randomized controlled trial] would have led to the demise of two potentially promising interventions.” While randomized controlled trials can provide much needed evidence about whether a program or intervention is working as expected, in complex systems change work, a more valuable question might be around why, how, and under what conditions a new intervention is showing promise.

I’m excited to see that developmental evaluation, along with its focus on helping organizations learn from, strengthen, and improve their work, has officially arrived in India. For more information, you can read our white paper on Evaluating Social Innovation or check out our webinar on the topic.

We also recognized that a relatively new evaluation approach, Developmental Evaluation, also seemed to be on everyone’s mind that week. “DE” as it is commonly known, is an approach to evaluation that is particularly well-suited for evaluating innovative programs, interventions, and initiatives many of which exist in complex environments, where the path to success is unpredictable. Conversations about developmental evaluation showed up in more than 30 sessions, in the hallways, and even at one of the nearby watering holes (The Local was an FSG favorite!).

Topics discussed ranged from how the role of the developmental evaluator differs from that of a formative or summative evaluator, to challenges associated with ensuring evaluation quality in innovative and complex situations, to what developmental evaluation is like in practice.

A take-away from our AEA experience was that an increasing number of evaluators and funders are intrigued by what developmental evaluation offers those who are trying new and novel ways of addressing complex social problems.

Outside of AEA, discussions around developmental evaluation also have spread like wildfire. Tamarack Institute for Community Engagement’s monthly e-magazine recently featured FSG’s white paper, Evaluating Social Innovation, highlighting the need for evaluation approaches that strengthen strategic learning, rather than solely focus on accountability for outcomes. In addition, after reading the Evaluating Social Innovation white paper, Steve Wright, VP Poverty Tools and Insights at the Grameen Foundation recently wrote:

"In our work at Grameen Foundation we have embraced the ideas of the [Lean] Startup which to me means that as we build solutions we are constantly testing our guesses against the reality that our customers are experiencing. This means that as we receive data from our customers we are constantly asking ourselves, "should we pivot or persevere?" The problem with this paradigm is that it flies in the face of traditional funder reporting where specific targets are set as deliverables. This might work great for an existing model that has a proven track record but how do we deal with new models where the specifics of the targets are unclear? This article describes a process called developmental evaluation (DE) which is specifically designed to help entrepreneurs use real-time evaluation insights to help refine the work."

To continue our collective learning about developmental evaluation, FSG will be co-hosting a webinar on Evaluating Social Innovation: Insights from the Field with the Center for Evaluation Innovation on November 14th. Webinar panelist, Meg Long, Deputy Director of OMG Center for Collaborative Learning, recently blogged about the power of using DE to fuel a research and development (R&D) approach to solving social problems. Meg will be joined on the panel by John Cawley, Director of Programs and Operations at J.W. McConnell Family Foundation. Wednesday’s webinar is another opportunity to hear from DE funders and practitioners about this emergent evaluation approach.

There are also many developmental evaluation related resources available through the AEA e-library, which can be perused at your leisure. 

We hope you, too, will join the DE conversation and tweet along with hashtag #strategiceval.

Advanced business analytics are now available (e.g. Hadoop) to help pore through quantitative AND qualitative data in seconds, generating insights and trends that would have taken days to gather. At a recent conference in San Francisco organized by the Economist magazine, one heard a few new terms being thrown around to describe this new-found superpower such as:

• machine intelligence, or the ability of computers to learn through analytics and get smarter with time
• reality mining, or the ability to mine digital data to explore trends and understand behavior
• nowcasting, similar to reality mining, a way to understand current trends as they are playing out in real-time

So how is the social sector benefiting from big data? One handicap that is familiar to all of us in the social sector is that data systems and information markets are, to put it mildly, under-developed. Schools struggle to properly keep track of the movement of teachers and students as they enter and leave the school system, leave alone data about their performance. Hospitals have difficulty capturing indicators such as re-admission rates that they are now required to report. Nonprofits may collect data to understand the characteristics and needs of the people that they reach through direct services, but often lack capacity to collect information related to long-term outcomes, such as changes in attitudes or behavior. In this context, the notion of big data and predictive analytics seems like a faraway dream.

• For example, big data is generated as formerly non-digital processes become digitized. Think everything from credit card payments to downloading mp3s to online shopping. As clicks replace bricks, they also generate data that can be aggregated and sorted at the touch of a button. Now think of the possibilities, say in a classroom, as blended learning becomes the norm and every click that that child makes in her journey toward mastery of a content area is recorded and available for analysis. The dream begins to get a little more real. 
• Big data is also generated as devices become more integrated into our everyday lives. We are almost immune to the devices such as sensors, cameras and GPS’ that play a role in our everyday lives. These devices are capturing information about us that can reveal key trends. Cut to the social sector and we see some of this technology being incorporated into healthcare. We now see virtual communities abound engaging in conversations about their health, driven by data from devices. Everyone from sleep apnea sufferers sharing data from pressure sensors in their masks, to runners sharing their data from insole sensors in their shoes is making big data a reality in healthcare. 
• Finally, we generate lots of big data as we willingly share information with the world through social media. Consider the billions of tweets that are sent out every day. Or check-ins on foursquare. Or updates on facebook.  Everywhere we go on social media, we leave “digital breadcrumbs” that point to collective trends. From Hollywood movie studios trying to gauge buzz around a new release to government intelligence agencies picking up chatter on potential flash points, analyzing social media posts is the new parlor game. The social sector isn’t far behind. Ushahidi is an innovative organization that uses aggregated trends from text messages and social media feeds to coordinate immediate response to natural disasters such as the Haiti earthquake and the Japan Tsunami.

So what’s possible now? Before we claim big data as the social sector’s new messiah, we should keep in mind that this phenomenon is still brand new and we need to address several issues around privacy, accuracy, and reliability before we can truly move forward. The use of big data in strategic learning and decision making is still quite nascent. Traditional data collection methods such as surveys, interviews and focus groups will continue to have relevance, as they help us go deeper, understand context and test insights. But there is no denying that the big data revolution has indeed begun! How is your organization using big data to change the world? 

With the "learning officer" title, I imagined relatives wondering if I were some version of a British bobby, blowing my whistle at people who refused to learn or didn't learn fast enough.

Of course, "chief learning officer" may be viewed as a euphemism for "director of evaluation," a more easily understood title. Yet when it comes to inspiring the creative, dynamic learning we all desire, evaluation director is not exactly the most welcoming title. Chekov believed that if an author placed a gun in the first act to create tension, at some point the gun had to go off. Evaluation is similar. It's a loaded gun and everyone waits for it to go off. It inspires fear, not inspiration.

I've come to embrace the "learning officer" title, for while mysterious, it is less threatening and, more importantly, it gives me more room to maneuver. I've come to believe that as a learning officer I have to be active in a range of activities— evaluation, strategy and organizational development.

By organizational development I refer to how a foundation strives to become a learning organization. Evaluative feedback is like water. When it flows over hard ground, it doesn't sink in. When it reaches soft ground, it feeds the soil and supports growth. Many organizations are like hard ground. They are torn by unresolved conflict, distrust, and unhealthy internal competition. These organizations have little capacity to absorb evaluative feedback and to adapt. To enable a whole organization to learn and evolve, we must soften the ground by building a staff environment that supports trust, collaboration, and inquiry. In an organization with prepared ground, evaluation matters.

Key to becoming a learning organization is the regular use of the pause button. Organizations have to take periodic time outs in order to absorb feedback, to reflect and to adapt. And these time-outs must be built into the master calendar. Otherwise it's too easy to allow the pressures of grant deadlines and day-to-day management to consume all of our time and energy, and to leave us on auto-pilot, even if that pilot is flying us into a cliff!

Here at The California Endowment, we've instituted the pause button in a couple ways. We bring our program staff together quarterly for a two-day in-person meeting to work on cross-department collaboration and strategy development. And we convene the local evaluators in our place-based work every few months to develop and refine the assessment tools we are using to measure progress.

In using the pause button and in creating a safe space for reflection, we give up some control and predictability. In our staff sessions, for example, we've seen the disconnects in our strategy bubble up and demand attention. In the daily grind, we tend to push the truly difficult issues to the margins but once we pause, we can't as easily escape them. They surround us!

In the ideal setting, synergy exists between evaluation and organizational development. The two modes interact in such a way to build an organizational culture that is dynamic and truly mission-driven.

In this sense you might say that a "chief learning officer" may also need to serve as a "staff culture developer."

Try describing that job to Aunt Shirley over a Thanksgiving dinner.

Jim Keddy is a Vice President and Chief Learning Officer at The California Endowment. His work is informed by a variety of sources and experiences including community organizing, public opinion research, Paulo Freire-inspired adult education, liberation theology and most importantly, basketball coaching with 7th and 8th grade girls.

Many of us are familiar with the parable of the “blind men and the elephant”. In various versions of the story, a group of blind men touch an elephant to learn what it is like. Each one feels a different body part, but only one part, and comes up with an explanation. For example, the one who feels the trunk claims that the elephant is like a tree branch, while the one who feels the tail swears that the elephant is just like a rope. Others go with pillar (legs), wall (abdomen), hand fan (ear), and spear (tusk). This parable has been used to illustrate several lessons, but the one that appeals to me the most, of course, is what it means for evaluation.

In evaluation, we often fail to see the whole elephant for various reasons. Traditionally, the field of evaluation has been led by a “reductionist” view of how the world works. We attempt to break complex phenomenon down into neat boxes and arrows, isolate variables, control for factors, and largely draw from Newtonian models of cause-effect and directionality. More recently however, the field has become acutely aware of the limitations of the traditional approach as the following tenets of complex systems that we work in have become more and more apparent:

1. Everything is connected; hence what happens in one part of the system affects another. For example, evaluation of a school improvement initiative may have to examine not just what takes place inside the school system, but also what happens in the broader community outside.
2. Cause and effect is not a linear, one-directional process; it is much more iterative. Does improving health of families improve their economic productivity or the other way around? The answer is probably both. We are increasingly learning that honing in on causation and attempting attribution is a herculean task, as there aren’t clear and straightforward links.
3. Context matters; a lot! What used to be considered “noise” in models of social change is now recognized as a core factor that can make or break an intervention. Teachers in schools with supportive conditions, for example, are shown to be performing at higher levels than those (even highly qualified ones) in other non-supportive schools.

What can we do?

So what do we, as evaluators and social change practitioners, do to be more cognizant of the “whole elephant”? How can we move from evaluating tusks and tails to really taking the entire pachyderm into consideration? Here are a few ways:

1. Create evaluation and learning systems, not just stand-alone evaluations: Sound evaluation and learning systems articulate what needs to be evaluated, when, how, by whom, with what resources, etc. in ways that will enhance learning throughout the organization and ensure that evaluation resources are spent effectively to boost organizational effectiveness.
2. Move towards shared, rather than just individual, measures of success: In an increasingly inter-connected world, it becomes imperative for organizations to work together to tackle complex and chronic social problems. Evaluation serves this scenario best when the measures used to track progress are common, shared and transparent. We can draw from examples of successful shared measurement systems.
3. Use innovative evaluation approaches that recognize complexity: The traditional paradigm of formative evaluation (to improve an approach or model) and summative evaluation (to prove that the approach or model works) may not quite take into account the complex and emergent nature of social change. Hence, other innovative approaches such as developmental evaluation, social network analysis, and human systems dynamics may be more suitable for certain interventions or at certain times in an intervention’s lifecycle.

At the same time, social change funders and actors are taking on increased accountability for results. They care about measuring inputs, outcomes and ultimately impact. They want to know if their strategies are working and they want to quantify the exact impact they’ve had. This is also a powerful trend.

However, when these two trends come together (complex problem solving and a thirst for monitoring and evaluation) they often result in a toxic environment that stifles innovation. Why is that?

The reason, as explained in FSG’s new report “Evaluating Social Innovation,”is that social innovation requires a different approach to evaluation.

Imagine the following scenario. You have decided to tackle youth unemployment in your hometown. You know that there are numerous root causes, dozens of actors that need to contribute to the problem solving process and a wide range of external factors that you can’t control, including elections, the economy, the revolving door of business and civil society leaders and the egos of all involved. It is hard to tell what success will look like; maybe getting the right actors to agree to work together systematically is already a big win?

Great news: a foundation is willing to fund your efforts, but with the following stipulations. You need to achieve impact within exactly 24 months. You need to define (“smart”) outcomes now, preferably predicting how many youths each quarter will gain employment based on your efforts. Exactly 12 months in, you need to demonstrate mid-point progress toward those pre-defined outcomes.

The illustration below, taken from “A Developmental Evaluation Primer” shows this phenomenon graphically.

The foundation wants your efforts to look like this (linear and predictable) and will evaluate you accordingly:

However since in reality you’re going after town-wide systems change (and not developing yet another job skills training center), your efforts will unfold more like this:

The mid-point evaluation of this effort would show that hardly any progress had been made and might conclude that your effort had failed and should not receive further funding.

However, rather than assessing exactly how far the effort is on achieving (pre-maturely defined) outcomes, more interesting inquiry includes:

• Can we discern the critical junctures and factors that have resulted in the spikes and troughs?
• We came pretty close to solving the problem a few times… what happened?
• How has the effort shifted course to accommodate learnings along the way?
• To what extent were the initial hypotheses going into the effort correct?
• What have we learned about the actors and external factors that play an influential role?
• How should the strategy change going forward?

Developmental evaluation, described in more detail in FSG’s new report, is focused on answering these types of questions as the effort unfolds. It does not provide a point in time snapshot of the exact outcomes achieved, but rather, it does justice to the unpredictable and creative ways social innovation unfolds and allows funders and actors to learn, adapt, challenge their assumptions, make decisions and improve their strategies real-time.

If you are funding or implementing solutions that follow the pattern depicted in Figure 2 above, then Developmental Evaluation might just be for you!

Our own experience working with successful learning organizations suggests that leaders can make a big difference in building the conditions for ongoing learning and improvement. Our conversation highlighted three key roles leaders can play in helping their organization maximize their learning from evaluation efforts:

1. Show a genuine commitment to evaluation

Leaders have a unique opportunity to communicate the importance of evaluation to both staff and partners and to show genuine committed to evaluation for learning. Jane Mosley, Chief Evaluation Officer with the Health Care Foundation of Greater Kansas, who is quoted in the GEO report, argues: “Unless evaluation and learning are made a priority and supported by organization leadership, they won’t be prioritized throughout the organization.” In my experience, foundation leaders effectively become evaluation champions when they embrace evaluation for learning in their day to day conversations with staff and grantees. When leaders start saying things such as, “Let’s plan to get together so we can learn more about this,” or “How can we get more information so that we can use the best data to inform this decision?” These conversations send a powerful message to the staff that shows leadership's real commitment to the use of evaluation for learning purposes.

2. Embrace failure

It is very important for organizations to learn from their mistakes and capture learnings that will help them achieve better results next time. Leaders are best positioned to create this culture where learning from what went wrong is accepted and encouraged. But this exercise is easier said than done “We’re in a field that fears failure. We feel that we always need to be the experts,” said Kathy Reich, Director of Organizational Effectiveness with the David and Lucile Packard Foundation, in the GEO report. As my colleague, Katelyn Mack, recently wrote about in a blog post, “admitting failure is scary, but it’s often a necessary part of learning, adaptation, and improvement. Leaders need to model this behavior for staff by admitting their own mistakes and explaining how they’ve learned from them.”

3. Provide the resources necessary for learning

Finally, it is important to acknowledge that even though leaders are key, it takes commitment across the organization to ensure that evaluation enables learning. You can’t have a learning organization if the staff doesn’t see the benefit and impact of collecting information and using it to inform their work. I want to leave you with a video from a presentation I made during the last GEO conference in Seattle, WA. The presentation talks about my experience trying to make my father’s farm a learning organization and touches on this last point of engaging the whole organization in learning.

Please feel free to leave comments or questions, and thank you for reading!

The Greater Cincinnati area has a strong history of collaboration and innovation.  Leading funders such as the Greater Cincinnati Foundation and the United Way of Greater Cincinnati wanted to further build momentum to advance collective efforts by focusing on shared measurement.  They saw opportunities in multiple sectors and wanted to assess the readiness of five key sectors for additional collaboration. 

Some sectors, such as education, had made great progress as described in FSG’s “Channeling Change: Making Collective Impact Work” (2012).  Other sectors were beginning their journey and building momentum.  The arts sector was one such example.  A leading organization in the sector is ArtsWave.  ArtsWave is one of the oldest and largest arts support organizations in the country.  It serves as a funder for over 100 arts organizations in the region as well as an advocate on behalf of the arts.

In 2011, ArtsWave designed what they called an “impact agenda” that included a limited set of desired outcomes for arts organizations as well as ultimate beneficiaries – the people of Greater Cincinnati.  Outcomes included a more connected and engaged community, improved quality of life for key demographic groups, and employment in specific industry sectors.  During our shared outcomes project with the five Greater Cincinnati sectors, FSG developed a “roadmap” for shared measurement (below). 

Roadmap for developing a shared measurement system

Many of the lessons learned from this project aligned with other reports of best practices for developing measurement systems.  For example, from an account of neighborhood revitalization in a mid-Atlantic region,  similar lessons learned included 1)  establishing or maintaining trusting relationships, 2) identifying critical support organizations, and 3) using data for continuous learning, not solely for reporting or accountability purposes.  Using data to inform practice and improve programs was also commonly noted.  Grantmakers for Effective Organizations has also distilled essential elements of useful evaluation.

But what was new in our project and different than past efforts?  Our project highlighted the importance of ways to facilitate conversations across entire sectors regarding shared outcomes.  New lessons learned from working across numerous stakeholders included: 1) scoping the boundaries of a sector, 2) identifying appropriate participants, and 3) establishing advisory or governance groups with clearer expectations for involvement and responsibilities.  My FSG colleagues have blogged about similar themes in the Greater Cincinnati context.  Future observations and project experiences should identify what other principles and tools apply to sector-wide social change efforts.

Finally, one really important lesson that we learned was to “start where you’re at.”  Data and outcomes are often more concrete topics of conversation that can lead to other important themes needed for community transformation.  Taking a first step helps build momentum to move down the path of social change.  As the work in Greater Cincinnati unfolds, we’re all looking forward to seeing how sector-wide change can be catalyzed by using data collectively to track progress and inform practice.

And yet many traditional approaches to evaluation aren’t effective when it comes to understanding what drives social innovation. A new report, released today by FSG and the Center for Evaluation Innovation, explores the ways many common evaluation approaches constrain innovation, for example, by trying to test a model too early in its development, and fixating on predetermined plans and original metrics that don’t evolve in response to the dynamic context.

Download Evaluating Social Innovation.

This topic gets raised in relatively closed conversations among funders, but public statements of errs in judgment, planning, and execution are still rare. In 2010, Robert Wood Johnson Foundation (RWJF) published a series of articles on programs that “didn’t work out as expected” in their Anthology: To Improve Health and Health Care. RWJF is among a select group of grantmakers that talk publicly about failure (whether they decide to use the word or not) and try to learn from them. Robert Hughes, one of the Anthology authors, further examines failure as a “Key to Foundation Effectiveness” in a series of blog posts on the topic. And recently, the field has engaged each other in dialogue about failure in a new way, with organizations hosting “fail fairs,” such as the one hosted by USAID Microlinks at the World Bank in DC.

What I’ve come to realize in reading accounts of foundation failures is the important role evaluation plays in identifying failure, diagnosing its cause, and making corrections (when feasible and appropriate). Three questions that confront failure head-on can improve how foundations and nonprofits understand, learn from, and use failure to be more effective.

Are we achieving what we set out to do?

Identifying failure begins by asking a simple question: Are we achieving what we set out to do? It takes courage to ask that question and answer it honestly.

Evaluators are well-equipped to help answer this question. We work with organizations to refine their strategies and increase clarity around their goals and the outcomes of their work. Then we use what we’ve learned to design and carry out an evaluation that helps us answer whether the organization is actually achieving what it has set out to do. For some organizations this requires a simple monitoring or performance measurement system that utilizes routinely collected metrics (e.g., blood pressure monitoring in health care settings), for others it requires a mixed methods approach that utilizes numerous different data collection activities (e.g., population-based surveys, interviews with key stakeholders).

Why isn’t it working?

Diagnosing the reasons for failure is equally as important as identifying that something isn’t working. Once you understand what the problem is, you are in a much better position to address it.

Again, evaluators plays an important role in helping organizations better understand the root cause(s) of an issue. We conduct interviews, observe programs, and do environmental scans to gather information about why an effort is (or is not) making progress. David Colby, Stephen Isaacs, and Robert Hughes have provided four reasons why programs do not succeed:

• Strategy or design flaws
• Challenging environments
• Faulty execution
• The inability to adapt in a timely fashion

These categories are helpful to consider (in the context of gathering evaluative information through a planned, systematic process) when diagnosing the reason(s) for why an effort is not working out as expected.

What should we do next?

It might seem obvious that the next step after identifying something that is not working and figuring out why is deciding what to do about it. There is also a tendency, however, to deny, dismiss, or avoid any evidence of failure. Especially in the social sector, to say a decent program is “failing” can be hard to accept. RWJF Anthology authors David Colby and Stephen Isaacs remind us, “Even programs that do not achieve their overall goals can have positive effects on the people they touch.”

While in some cases failure does mean the end of a program or initiative, there are numerous examples of when failure was just a short detour on the road to success. Learning from mistakes, so that programs can be improved is essential to “constructive failure” in philanthropy.

It's no accident that these questions are phrased as though the work is ongoing. Learning that a program has or hasn’t worked shouldn’t just happen at the end of a program. It’s exciting to work with foundations that are interested in understanding what’s really taking place as an initiative unfolds. By asking these questions throughout the process from a position of genuine learning, evaluators and key decision makers can make the changes necessary to get a program back on course. This doesn’t guarantee, of course, that the ultimate goals will be achieved. Nonetheless, it provides an important opportunity for mid-course corrections that could mean the difference between success and failure.

So, if you’ve failed… Admit it! Share what you’ve learned with others, so they can grow in their knowledge of what it takes to make a difference. As Tessie San Martin, President/CEO of Plan International USA, bravely wrote in a blog post following the World Bank Fail Fair, “We do not celebrate failure often enough. But we should. …We are failing. And in that failure we are learning, adapting and advancing, and therefore improving our ability to improve the lives of children around the world.” When realized in the context of learning, failure isn’t the end of success. It’s the beginning.

Measuring results is hard. And even harder when evaluating “impact” – aka “change”. So we often settle for proxies that are more pragmatic. Things we can count. However, more and more I see this as a dangerous method in the long-run.

Take the aforementioned digital divide. If we articulate the problem only as “bridging the digital divide” – aka ensuring “access to information technology” we’ve done ourselves a favor as access is somewhat easy to count. And according to Jeff, “To be clear, for a long time, there was a problem with the digital divide with low income households and students of color unable to participate in the digital age, and the problem was ensuring access. So at that time, measuring access was actually okay. But of course, once you ensure access, you have to ensure that use is of quality.”

In other words, as you get closer to the finish line, you better move the goal post again. Because if we stick to the original measures of success around access or usage (screen time) as a proxy for success, we’ve gone down a dangerous path. As you can surely imagine, more screen time is not necessarily good! It hampers creativity, interpersonal interaction, self-directed exploration, etc. The problem should have been articulated differently: “low income children aren’t able to experience the personal and academic benefits that can come from access to information technology.” Then the measurement would not have been number of children reached, but the actual personal and academic benefits. These benefits could include improved educational achievement, ability to lead healthier lives, increased economic opportunity, and participation in their communities Harder to measure of course, but avoids the trap of declaring success just by posting high usage numbers.

Microcredit is a similar story. The problem was wrongly articulated as “not enough low income households can access loans.” Voila – the measure of success becomes number of loans or volume of loans dispersed. Easy to count and easy to celebrate as a win. We all know what happened in this case. Instead, the problem should have been defined as “not enough entrepreneurs can access capital to help grow their businesses or farms”. In this case, the measure of success would have been growth in asset base and cashflow of borrowers over time. And loans would only have been given to those who could put them to productive use. Again, by settling for a proxy (number of loans disbursed) impact was in many places quite negative.

Similarly, the US mortgage crisis comes to mind. The problem was unfortunately defined as “not enough low income households own their own home” rather than “not enough low income households are building a long-term asset base through home ownership”. Subtle difference in articulation, but major implication on what counts as success.

In all of these examples, focusing on increasing outputs of the (wrongly defined) problem actually ended up having a negative effect on impact!

As you think about your own theories of change, ask yourself if you’ve correctly defined the ultimate problem you’re trying to solve and if any of the indicators you’re tracking along the way may in the end be counterproductive.

The aim of shared measurement is to focus different stakeholders’ attention on a set of common measures as a way to bring alignment, focus, and increased impact to challenging social problems.  For example, increasing high school graduation is a commonly agreed-upon indicator of improved educational outcomes.   I have seen the process of developing shared measures play out in the context of the entire education continuum, as part of a “cradle-to-college” initiative in the Seattle region.  As detailed in an earlier blog post by my colleague Fay Hanleybrown, identifying shared measures is an import part of a complex and multi-part process.  The steps that follow defining common measures are as important as identifying the measures themselves, as these steps ensure that the data is reported, accessible and useful, which is the key to supporting data-driven, large-scale change.  These steps include: 

• Assess if, where, and how the data are collected: Once shared metrics are identified, the first step is to understand if and how the data are collected.  It is important to go through the process of mapping out the data source, frequency of collection and reporting, how the data are disaggregated (e.g. by race/ethnicity, income), whether the data are publicly available, or if obtaining the data would require a data sharing agreement.  For example, in Washington State, data on high school graduation are publicly available from the Office of Superintendent of Public Instruction, are reported by district and school, and are disaggregated by race, income, and other student sub-groups.
• Determine how the data will be used: Once the source for each metric has been identified, it is important to consider how the data will be used.  For example, with high school graduation, data could be used to track the progress toward a specific goal or target – such as increasing the on-time graduation rate to 90%.  Collecting annual graduation data over time for different schools in a district would show which schools have met or are approaching the goal, and where progress has not been made or ground has been lost.  Data can also be used to inform practice.  This would mean looking at high school graduation data over time to identify those schools that have seen progress in increasing their graduation rate to explore “under the hood” to understand how and why this progress has been made.  Ultimately, the practices of those schools that have improved could be shared with schools that have seen less progress. 
• Where needed, build relationships to collect data: For those data that aren’t readily available through public data sources, relationships are key.  Data sharing agreements may be required to get access to the needed data, which requires support from various stakeholders.  Developing trust and understanding of how data will be used are critical for getting access to data and creating buy-in for the work.  For example, a college mentoring program could pursue a data-sharing agreement with a school or district to understand how the students they are serve are doing relative to graduation requirements and if their students are on-track to graduate.  Access to this level of data – which is more specific than whole-district or school-level data – often requires the execution of a data sharing agreement to govern how data can be shared, stored, and used.
• Make data accessible and relevant:  Once the data are collected, it is important to make sure the data are useful, relevant, and digestible to the target audience(s).  To track progress of a collective impact initiative, for example, it is useful to display data relative to a time-bound goal or target.  Displaying a district’s graduation rates of 74% in 2010 and 78% in 2011 alongside a goal rate of 90% in 2018 demonstrates how that district is performing relative to a specific target, and shows how much progress is still needed.  To improve practice, providing more granular data is necessary to inform decision-making.  For a college mentoring program, for example, it would be useful to provide reports on which students are and are not on-track relative to high school graduation requirements.   This would provide specific information about which students need what type of support to be able to graduate, and could help the program to better target its attention and support.

Identifying shared measures is an important part of getting participants in collective impact initiatives on the same page and aligned in their thinking about what they aim to collectively accomplish.  But it’s only the beginning!  With thoughtful consideration of each of these steps in the process, data on these metrics can ultimately be collected, analyzed and used to drive further progress. 

Where have you seen shared measures effectively adopted and used to solve complex problems?

For decades, the prevailing answer has been that foundations should conduct the social equivalent of research and development. Pilot innovative new programs, test them to see if they work and, once a solution has been discovered, let governments or others with larger resources fund the solution at scale. This is a tempting approach for foundations – it encourages them to continually test exciting new ideas, each requiring only a small amount of funding, and it holds out the promise taking credit for an important breakthrough. Under this paradigm, the familiar role of evaluation is to isolate the impact of a specific program, test its effect on a problem, and render a judgment as to whether it works or not.

The only problem is that this paradigm rarely – if ever – works. Social problems are complex. They are created and perpetuated by a wide range of factors beyond the control of any single organization. Even if a promising new approach is discovered, countervailing political factions may block its adoption, more funding may not be available, or the necessary management talent may be scarce.

In recent years, therefore, foundations have experimented with new paradigms of social impact: Funders that practice venture philanthropy take direct responsibility for growing effective organizations to scale. Others choose a catalytic approach, impacting an issue by leveraging non-monetary tools to shift government or corporate policies, or by motivating changes in individual behavior. More recently still, foundations have begun to pursue collective impact, working to align and coordinate the efforts of many different private, civil, and governmental organizations toward a unified plan of action. Each of these new approaches to social impact defines a different role for evaluation. The venture philanthropist needs measures of organizational effectiveness and sustainability. The catalytic philanthropist needs ways to understand the overall problem, rather than any individual grant. And the collective impact instigator needs a shared measurement system that enables numerous organizations to track and compare their progress. As a result, the field of evaluation today is in continual ferment as many new methodologies are being tried. What all of these new approaches have in common is a focus on learning so that the funders themselves can become more effective over time.

Yet old paradigms die hard. Many funders still adhere to an evaluation approach that aims to test pilot programs to prove whether or not they work, discounting other approaches as lacking scientific validity. What matters, of course, is not whether any one approach to evaluation is right or wrong, but whether it fits the underlying theory of change.

The time when a single approach to evaluation served all funders is gone forever. It has been replaced by an outpouring of innovation in evaluation that is showing foundations new ways of achieving impact, such as the use of developmental evaluation to evaluate new, innovative approaches to social change. And that is a very good thing.

What is learning?

1. Individual, group, organizational: refers to who learns. One of my favorite definitions of learning is, “acquisition of skill or know-how, and the acquisition of know-why.”
2. Informal/Formal: this reflects the how of learning – informal learning is typically unstructured and experiential. Studies have shown that 70-80% of what people know about their jobs, they have learned informally from the people with whom they work. Formal learning is where the content has been chosen by others and presented to the learner, such as in a training program.
3. Intentional/Unintentional: reflects whether the learning experience is thought-through and planned (intentional), or if learning happens through daily experience and/or is serendipitous (unintentional).

What does it take to learn within an organization?

I believe learning requires consistent engagement in 5 key learning processes:

1. Engaging in Reflection - creating space, slowing down, paying attention, creating new patterns of thinking, creating alternative interpretations, creating new theories of action
2. Engaging to Dialogue - participants working together toward common understanding, finding common ground, re-examining all positions, admitting that others’ thinking can improve on one’s own, searching for strengths and value in others’ positions, listening to understand
3. Asking Questions – seeking clarification, probing assumptions, reasons and evidence, illuminating viewpoints and perspectives, probing implications and consequences, questioning the questions (Socratic questions)
4. Identifying and Challenging Values, Assumptions and Beliefs – asking questions (testing assumptions), surfacing mental models, seeking evidence, understanding inferences
5. Seeking Feedback –asking for and providing feedback on experiences, assumptions, perceptions, and actions

What is the relationship between evaluation and learning?

“Evaluation is a core learning practice. It provides the content of learning as grantmakers and their grantees explore the results of their work and how to improve their performance…Evaluation, of course, is not the only way in which organizations learn. But grantmakers must think evaluatively about their work and have access to the information, feedback and data that only evaluation can deliver.” Evaluation in Philanthropy: Perspectives from The Field, GEO, 2009, p. 6

You see, all sessions in the area of evaluation are framed to be about “Evaluation and Learning”. Specifically to “help grantmakers and their partners better use evaluation to foster learning and make real-time improvements in their work.” Foster learning. Real-time improvements. Music to my ears.

33. Number of times “evaluation” appears in the program. 42. Number of times “learning” appears in the program. 12. The number of times “improve” shows up. 3. The number of times “prove” shows up. Granted, this is not a very scientific analysis of the program – but directionally it is clear that evaluation is not framed as a practice that grantmakers should pursue to “prove” something or to pinpoint and quantify specific impact, or to impose on grantees. But rather, to learn and improve.

Specifically, I wish I could be a fly on the wall at the following sessions:

• Building Capacity for Evaluation and Learning (featuring Innovation Network): High-performing nonprofit organizations seek and use data and feedback to continually assess and improve their work. And, behind these efforts are supportive grantmakers who embrace the role they can play in helping grantees make effective use of information.

• Learning in Public (designed by and featuring the The David and Lucile Packard Foundation): Foundations spend millions of dollars each year on evaluation and learning activities, yet they rarely make evaluation results public. And often foundations make decisions based on these results without consulting key stakeholders or asking for their help in interpreting evaluation findings. There is an alternate approach emerging: sharing what an organization learns while it is in the process of learning — “learning in public.”

• Practical Evaluation in Complex Communities (designed by and featuring Jessica Bearman): During this lively session, speakers and participants will explore the top challenges inherent in evaluating place-based grantmaking — and discuss practical strategies that have worked to engage communities and colleagues, share data and results and embrace the complexity of real world efforts.

• The Hard Truth About Strategic Learning in Five Minutes (designed by and featuring the Center for Evaluation Innovation): Many foundations are trying to incorporate strategic learning into their grantmaking strategies, particularly strategic philanthropists who recognize that complex problems require dynamic and transformative solutions. Strategic learning promises that lessons that emerge from evaluation and other data sources will be timely, actionable, and forward-looking, and that strategists will gain insights that help them make their next move in a way that increases their likelihood of success.

• Reinventing Evaluation for Social Innovation and Change (designed by and featuring our very own Hallie Preskill): It seems funders are still struggling to find an evaluation approach that is well suited for social innovation in complex environments. In this session, participants will engage in learning, reflection and dialogue about “developmental evaluation,” the experiences of grantmakers who are implementing the approach in their work and the ways in which it could add value to their current evaluation portfolio.

• Leadership and Learning: Two Sides of the Same Coin (designed by and featuring Fourth Quadrant Partners): Being a learning organization is not easy, but it does not need to be as complicated as we make it. We make learning complicated by seeing it as something separate or something that takes precious time away from the task at hand. By adopting a set of tools and a philosophy about leading, we can strengthen our supervision, leadership, learning and self-evaluation abilities as well.

Increasingly organizations seem interested in developing such a system or framework; what we call a strategic learning and evaluation system. As we guide, collaborate, and co-develop these learning and evaluation systems with our clients, I am seeing an emergence of a new kind of thinking, which builds on what we, as evaluators, often call “thinking evaluatively. The emergent thinking includes an element of strategy, and I have started fondly calling this "thinking strategevaluatively.”

Thinking evaluatively reflects an orientation toward asking useful and meaningful questions and using systematically collected data to act on findings. The Bruner Foundation has created an “Evaluative Thinking Assessment Tool” that defines evaluative thinking as,

“A type of reflective practice that incorporates use of systematically collected data to inform organizational decisions and other actions. The key components of evaluative thinking include:

• Asking questions of substance,
• Determining what data are needed to address the questions,
• Gathering appropriate data in systematic ways,
• Analyzing data and sharing results,
• Developing strategies to act on evaluation findings."

In addition to the key components listed above, thinking “strategevaluatively” reflects an ability to consider how evaluation activities can inform and improve an organization’s or key program area’s strategy.

Take, for example, a health care foundation desiring to increase insurance coverage in underserved areas. Evaluative thinking may lead foundation staff to ask questions such as, “What activities have been most effective in increasing enrollment in health insurance among our target population?” And to collect data that are used by program staff to tweak and improve specific enrollment campaigns or program activities, so that the foundation can meet its enrollment targets.

However, a similar question that would reflect thinking “strategevaluatively” may be, “What are the environmental factors - political, social, economic - that are helping or hindering our organization from reaching the enrollment targets?” This question connects the world of strategy and its implementation with evaluation practice. In the long haul, the answer to this question can help the foundation decide how the strategy may need to change given any new developments in the external environment.

One of the challenges that organizations may face in fostering “strategevaluative” thinking is the widespread division between those responsible for strategic planning and implementation and those responsible for evaluation. Using evaluation to inform strategy requires a culture shift in many organizations. Working at this intersection (described in a previous blog post by Hallie Preskill, FSG Managing Director) requires that program officers be equipped to think – and act – evaluatively in their work and demands that evaluation officers develop a sound understanding of strategy and its implementation.

Some foundations, including the The John S. and James L. Knight Foundation, have created a specific staff position that straddles strategy and evaluation. At Knight this position is held by Mayur Patel, Vice President of Strategy and Assessment. Creating a staff position with insight into both strategy and evaluation practice is one way to help strengthen the infrastructure that is at the intersection of strategy and evaluation. We have found that the process of developing a strategic learning and evaluation system also appears to support a culture shift that values and utilizes “strategevaluative” thinking, as well.

What examples do you have of organizations taking steps to strengthen the interplay between evaluation and strategy processes? We have more information on the components of strategic learning and evaluation systems here, or summary of our work with The MasterCard Foundation. Many more examples will be coming to our blog and website soon!

Two projects that exemplify these aspects in varying degrees are the Human Early Learning Partnership’s Early Development Instrument (EDI) at the University of British Columbia and the Kids Integrated Data System at the University of Pennsylvania. Both efforts have been underway for more than a decade. Significant time has been invested in designing and building these initiatives to operate at scale. These successful efforts that use data and measurement systems in the service of improving children’s lives challenge us to think about what is possible in often highly complex and under-resourced conditions.

It seems that if I tie my resolutions – running a half marathon, cooking more – to a guiding purpose, such as staying healthy and strong, that I might be more likely to stick with them over the next 365 days. When I’ve made resolutions that are action-oriented and tied to an explicit purpose, I have had more success than for those that are purpose-less.

Just as purpose can help us fulfill resolutions to spend more time with family and friends or quit a bad habit, it is also an essential component of good evaluation practice. In Michael Quinn Patton’s book, Utilization-Focused Evaluation, he describes how many evaluations are designed and conducted without considering its purpose or the evaluation’s “intended use.” Patton writes, “To evaluate how well you’re doing, you must have some place you’re trying to get to…For evaluators, this means clarifying intended uses of a particular evaluation.”

How evaluations are to be used and by whom is strongly related to their purpose. Patton discusses six primary types of purpose/intended use:

• Summative Judgment
• Learning and Formative Improvement
• Accountability
• Monitoring
• Development
• Knowledge Generation

For example, if an evaluation’s purpose is to better understand what works and what doesn’t (i.e. learning), then the primary use of evaluation findings is program improvement. If the purpose of the evaluation is to assess whether goals are being met and funds are being used as expected (i.e., accountability), then oversight and compliance are primary evaluation uses.

Who the evaluation stakeholders (i.e., intended users) are – program staff, foundation board members, community residents – can also influence the purpose and intended uses of the evaluation. Board members may expect the evaluation to serve a different purpose than the program beneficiaries. Check out our blog post, webinar, or FSG's report, Practical Guide for Engaging Stakeholders in Developing Evaluation Questions, if you’re interested in learning more about how to get stakeholders more engaged so that evaluation findings get used.

I invite you to reflect on ongoing or upcoming evaluations and take stock of the purpose and intended use of the evaluation. Have you explicitly articulated the evaluation purpose? How might the purpose have changed over time? What activities or process might facilitate use of your evaluation findings?

May your year be filled with purpose-filled and useful evaluations!

Lesson Learned – The concept “time is money” defines one of the fundamental differences between Mexican and American culture and affects the way evaluators interact when they are conducting interviews with people from Mexico. Time has high value in America, and evaluators in the US tend to be very concise and to the point when they are conducting interviews. However, Mexican interviewees might be working under a different assumption: “It’s better to have friends than money” (very popular saying in Mexico), and will start holding casual conversations to build sympathy and buy-in before delving into the topic at hand.

Hot tip – Don’t try to force your Latino interviewee to be direct when they are using indirect or circular language. Instead, let the interviewee talk about those seemingly unrelated topics and look for relevant information that can help you answer the interview questions. Remember that nothing is really unrelated. When necessary, use words that will narrow interviewees’ answers. (e.g., “What was the result of your interaction with the job agency? Did you find a job after visiting the job agency?”)

My first effort to implement strategic learning in an organization goes back to my time working at my father’s farm in Capilla de Guadalupe in Mexico. When I was 16 years old my father put me in charge of our milk production. Even though I grew up on a farm, I always had a fascination with computers, so I told my father that I would accept the job, but I needed to buy specialized software to help us manage the 200 cows we had on the farm. I was very excited to have a system that could tell us precisely when our cows needed to get pregnant, when to stop milking them, and so on. It took me some time to convince my father about the benefits of the system but after I showed him the impact it could have on the farm’s overall goal of producing more milk, he was fully on board. And, there I was, my very first time helping an organization generate relevant data that could be used for better decision-making!

As I reflect on my work as an evaluator at FSG, I find myself incorporating some of the things I learned from this experience at my father’s farm in my daily work. When we develop Strategic Learning and Evaluation systems at FSG we help our clients define a strategy that will help them achieve their goals, help them generate relevant information that can be used for decision making, and conduct interviews with leadership, staff, and other stakeholders to make sure that resulting evaluation findings meet the needs and expectations of the organization. Finally, we facilitate working sessions to ensure that staff understand and feel ownership of the new system, and are able to use the evaluation processes and findings to support individual, group and organizational learning.

Do you have a similar experience where you learned about evaluation in a seemingly unrelated project? Share it with us in the space below.

Through our work with the John S. and James L. Knight Foundation, we have started using social media and website analysis to evaluate impact. The Knight Foundation’s Community Information Challenge has supported more than eighty projects to create more informed and engaged communities. These projects are relying heavily on the use of websites and social media to reach and engage their target audiences, which may include youth, policymakers, and the general public, among others.

During our presentation, we highlighted the work of GrowWNY, a Community Information Challenge project that aims to engage and connect organizations promoting a healthier environment in Buffalo, NY. GrowWNY is using Facebook, Twitter, Flickr, YouTube, and a new website platform to reach and engage nonprofits, as well as the broader community. Then, in true “think tank” fashion, we asked session attendees to consider a variety of scenarios that included a description of a community engagement project, its goals, its program design and key online and offline activities, and a key evaluation question.

With the help of session attendees, we identified some outstanding questions and key insights about how social media and web analytics could be used to answer questions about who is being reach, how people are engaging, and whether dialogue among residents is growing.

Outstanding Questions:

• Using social media well takes up valuable time and resources. How can evaluators address the matter of which social media activities are getting the greatest return on time, resources being committed? 
• IP addresses can be unreliable. What tools are most effective in evaluating whether users are part of the project’s target audience, especially for place-based efforts? 
• How do evaluators approach privacy issues when using social media and web-based tools in their practice?

Participants generated several insights about evaluating community engagement using social media and web analytics, as well. Thankfully, some of these insights help to answer a few of the outstanding questions above!

Key Insights:

• Identifying who you are reaching online is a universal challenge. Nonprofits seeking to increase community engagement aim to reach people in a specific geographic location; knowing who their online audience is matters. When asking questions about program reach and engagement, there need to be mechanisms in place to identify online users and participants based on key characteristics (e.g., previous engagement, geographic area, demographics). Registration using a unique system or a Facebook-integrated system is one way of better understanding online reach.
• Social media generates qualitative information through comments and uploaded content that can be used to understand how and why users are engaging. Go beyond page views and counts to understand what people engaging in your program are saying – about their participation, your work, or each other. This could be done by reading and drawing out themes of comments on Facebook or in response to blog posts.
• Tools beyond social media, such as surveys, can help connect offline engagement with online engagement. They might be collected at an in-person event, distributed to an email list, or embedded online. Survey Monkey was mentioned as one tool that’s easy to use and integrates well online. 
• There is no one “right” method. A mixed methods approach is needed to evaluate behavior as complex as engagement. While social media and website analytics may help answer some questions, they often are insufficient and should be supplemented by other methods of data collection, as well.
• Consider barriers to online engagement. If your target audience doesn’t have access to the internet or can’t handle one more registration process, then you might have trouble getting people to log-on, comment, or more deeply engage online. These barriers also mean that who you reach online could be very different than who you reach in “real world,” offline activities.

The evaluation field can hold its own in any discussion of partisanship. Our field mastered that approach at a time when Congress was actually still pretty good about engaging across the aisle. The evaluation partisans? The randomistas and the story tellers. The randomistas, when not busy implementing their experimental studies, have been vigilant in their efforts to seek out and eradicate anecdotes. The story tellers have spent a lot of energy avoiding a step onto the slippery slope they envision if you admit that an RCT could sometimes make a contribution. With the two groups in the same room, it’s often not been pretty.

I want to introduce you to some bipartisan randomistas and story tellers. I want more of them.

Esther Duflo is Abdul Latif Jameel Professor of Poverty Alleviation and Development Economics in the Department of Economics at MIT. Together with Abhijit Banerjee and Sendhil Mullainathan of Harvard University, she founded the Abdul Latif Jameel Poverty Action Lab in 2003. She and Banerjee recently co-authored Poor Economics. As noted in the book’s overview, “Through a careful analysis of a very rich body of evidence, including the hundreds of randomized control trials that Banerjee and Duflo’s lab has pioneered, they show why the poor, despite having the same desires and abilities as anyone else, end up with entirely different lives.”

Yes, “hundreds of RCTS” – she is quite the randomista. But let’s take a look at three studies she and colleagues implemented in Kenya where rates of early child bearing and risky sexual behavior were alarmingly high among young adolescent girls. In particular, young women age 15 to 19 were found to be five times more likely to be infected with HIV than young men in the same age cohort, apparently because the young women had sex with older men who have comparably high infection rates. The first experiment tested the health education curriculum offered in schools; its focus is on abstinence until marriage and condoms are not discussed. The second strategy tested the provision of key information to the girls – the fact that older men are more likely to be infected with HIV than younger ones. The third experiment tested an effort to help girls remain in school by paying for the mandatory school uniform that many could not afford.

The studies showed that the curriculum did not increase knowledge about AIDS or reduce pregnancies. The “sugar daddies” experiment found reduced pregnancy rates mainly attributable to a reduction by two-thirds in pregnancies where an older male partner was involved. And for every three girls able to stay in school because of the free uniform, two delayed their first pregnancy. But this effect was seen only in schools where teachers had not been trained to deliver the curriculum. In those schools, there was no difference in pregnancies compared to girls in schools with no interventions

Now, read what the authors tell us about these findings and then I’ll tell you the two points I want to highlight:

“Putting these different results together, a coherent story starts to emerge. Girls in Kenya know perfectly well that unprotected sex leads to pregnancy. But if they think the prospective father will feel obliged to take care of them once they give birth to his child, getting pregnant may not be such a bad thing after all. In fact, for the girls who cannot afford a uniform and cannot stay in school, having a child and starting a family of her own could be a really attractive option…This makes older men more attractive partners than younger men who cannot afford to get married (at least when the girls don’t know that they are more likely to have HIV). Uniforms reduce fertility by giving girls the ability to stay in school, and thus a reason not to be pregnant. But the sex education program, because it discourages extramarital sex and promotes marriage, focuses the girls on finding a husband (who more or less has to be a sugar daddy), undoing the effects of the uniforms.”

What do I see here? I see RCTs being used in what is essentially developmental evaluation, a purpose and methodology that story tellers typically would not think of as a match. More important perhaps, the discussion of the findings (with the word “story” in the first sentence) highlights complexity, something that story tellers cite as a big limitation of RCTs- their perceived inability to contribute when issues are complex.

What about the story tellers? An article in the Stanford Social Innovation Review by Suzie Boss (“Amplifying Local Voices”) describes GlobalGiving’s storytelling project and the work of a UK-based firm, Cognitive Edge. As noted in the article “Listening to stories may seem simple, but turning this into a method for monitoring development work has meant drawing on fields as diverse as complexity theory, behavioral psychology, and technology.” The goals of the work include helping NGOs have more systematic data and to get to better results more quickly. For Cognitive Edge, the aim is not “gathering heartwarming stories for their emotional appeal” but analyzing large quantities of what they call micro-narratives. They have developed analysis software to reveal patterns as stories form clusters around particular topics.

Randomistas would say that one example of more rigorous story telling is not sufficient. I can offer two others you could check out: See Change Evaluation has an approach they call Story Science. And Most Significant Change is an increasingly known evaluation approach involving a comprehensive and structured process to systematically identify change, using stories.

What these story tellers have in common is an approach that is grounded in rigor and designed to produce systematic data. “Rigor” and “systematic” are words partisan randomistas assume do not ever apply to story tellers. And the earlier example about RCTS showed them being used for learning. Oh my. There’s a word - “learning”- that partisan story tellers never think of as a motivation for a randomista.

Hopefully, we are moving toward a time when the evaluation field will engage in more bipartisanship as we consider methods. Then we will put the bipartisan randomistas and story tellers to work on deficit reduction.

Education reform is sticky enough. What about the impact of credit-default swaps? And LIBOR manipulation? The decision to allow Bear Stearns and Lehman Brothers to go under? The impact of bailing out banks?

That’s tough stuff, complexity for sure. Then again, if evaluators cannot deliver value to society on the really big problems, what are all the meetings for?

Today, right at three years after Lehman filed for bankruptcy protection, it’s still not clear what happened, or what it all meant. The Wall Street Journal (Oct. 6, 2011) is reporting that small banks received $4 billion federal funds this year to stoke lending to small businesses. The notion was that the capital infusion would unlock credit for small businesses and, in turn, stimulate the economy.

But then the banks turned around and used more than half – $2.2 billion – to repay federal loans they received previously, through the Troubled Asset Relief Program.

Sorting this out is going to take a pretty robust disputatious community of truth seekers, to borrow Donald Campbell’s fierce imagery.

Then again, we’re a long way from Campbell’s Experimenting Society, which he imagined as “an honest society, committed to reality-testing, to self-criticism, to avoiding self-deception. It will say it like it is, face up to the facts, be undefensive and open in self-presentation.”

In the Experimenting Society, “there will be public access to the records on which social decisions are made. … There will be sufficient separation of governmental powers so that meaningful legal suits against the government are possible.”

Perhaps journalists will emerge as surrogate evaluators, carrying out at least a portion of the explanatory work.

Gretchen Morgenson and Joshua Rosner have made a valued contribution with Reckless Endangerment: How Outsized Ambition, Greed, and Corruption Led to Economic Armageddon (Times Books, 2011). As captivating as it is, this book touches only some of the strands of the complexity.

We need more folks like Kelly Creedon.

Based in Boston, Kelly is a documentarian and artist and multimedia producer, a description that may point to the need for a multimedia evaluator.

Kelly has devoted the last few years to documenting, explaining and sharing the foreclosure stories behind the complexity. Her multimedia project includes a traveling exhibit and an online presence: We Shall Not Be Moved.

Here is how Kelly describes We Shall Not Be Moved: “The project tells the story of a growing grassroots movement that is using the power of bringing people together to help keep families in their homes after foreclosure.”

That’s the boilerplate. It’s notable enough, in that not many folks stick around after foreclosure to document what happens then, how residents organize to defy the order, to hang on.

Beyond the boilerplate, there are the stories.

Listen to one. Listen to a few. Listen to Marshall Cooper.

Given all the dominoes that tumbled since 2008, all of the systems that failed, all of the warning bells that either didn’t ring or rang and rang and rang, all of the internal audits and external regulators, all of the planning and ongoing feedback loops and all of the post-hoc evaluation and, for goodness sake, all of the lessons learned (every report on a crisis must end with a list of “lessons learned”)….given all of these evaluative tools and evaluation resources, how does all the $**t running downhill land on Marshall Cooper?

The school described itself as: “High tech, high teach, high touch". One of the first of its kind, there was one computer for every two students. Given a great deal of latitude to experiment, and various state waivers, the school was teacher-led, based on a collaborative model, and focused on project-based learning activities. Other details include:

• 170 students in grades 4-8 (mixed-grade classes)
• The curriculum changed every 9 weeks based on student interests and needs 
• Two-tier staff structure - lead teacher and three associate teachers held year-round, higher-salaried positions; Others were "generalists," with regular nine-month appointments, lower salaries
• It was housed in a renovated downtown building - students could learn in the community
• 500 applications for 160 slots (originally grades 4-6; grades 7-8 added in year 2)
• Expectation was that many teachers would be interested in teaching at the school; in the end, had to search high and low for teachers

At the time, it made sense to conduct a formative evaluation; the teachers could use the evaluation findings to refine, adjust, and improve the program. The comprehensive evaluation was designed to address 18 key evaluation questions, and data were collected through interviews, surveys, tests, observation, focus groups, photographs, and document review. As was common practice in the field, 75-100 page year-end reports were written and delivered – in August, preceding the next school year.

It wasn’t long before the evaluation data uncovered serious problems in the school, including the following:

• The lead teachers didn’t like each another
• Students in grades 4-6 were developmentally very different from those in grades 7-8; this created unexpected instructional and disciplinary challenges 
• The school attracted more academically challenged students; the teachers were not prepared 
• The teachers described their world as a “speeding train out of control” 
• There were accusations of racism among the staff 
• They hired and fired three organization development consultants 
• Both local newspapers continuously wrote negative stories about the school 
• Parents starting withdrawing their children within a couple of years 
• By the end of the 4th year, the year-round teaching positions were discontinued; the three lead staff were transferred to other schools 
• By the end of the 5th year, the school was having trouble maintaining enrollment
• At the end of the 7th year, it became a traditional school 
• At the end of the 14th year, the school closed its doors

As the problems were becoming more and more visible and then entrenched, I became increasingly frustrated that the evaluation was not helping. The lengthy reports I delivered in August were just too late to be useful – the teachers thought they had already fixed the problems. And, though I did try to meet with the teachers to share what I was learning along the way, they perceived my evaluation role as the school’s “historian” and “objective observer.” As a result, the evaluation provided little value to them.

Over the years this experience significantly influenced my thinking, research, writing, and practice. What I’ve come to realize, is that The Saturn School was a complex solution to a complex problem, and that the formative evaluation was no match for the developmental, chaotic, emergent, unpredictable, relational, and dynamic nature of the school. Unfortunately, the evaluation had no mechanism or role for helping the teachers learn and adapt as they developed various aspects of the school.

What I came to realize was the need was another type of evaluation – Developmental Evaluation. However, at that time the idea of Developmental Evaluation (DE) was just being formulated by Michael Q. Patton, and had not yet resulted in any articles or books. Since then, Patton has published his book, Developmental Evaluation and has presented and written widely on the topic. He defines DE “as an approach to evaluation that is grounded in systems thinking and supports innovation by collecting and analyzing real time data in ways that lead to informed and ongoing decision making as part of the design, development, and implementation process.”

In an organic, learning-oriented, and adaptive way, Developmental Evaluation explores the following questions:

• What is developing or emerging as the innovation takes shape?
• How is the larger system or environment responding to the innovation? 
• What seems to be working and not working as the innovation unfolds? 
• How should the innovation be adapted in response to changing circumstances?

In this regard, a Developmental Evaluation is responsive to the emerging information needs of stakeholders as the initiative unfolds. The evaluation team provides continuous real-time feedback using multiple formats to inform strategy during the design, development, and implementation process, and ensures that learning processes are embedded throughout the evaluation.

With a grant from the Center for Evaluation Innovation, FSG has been studying potential uses and lessons learned from funders of Developmental Evaluations and evaluators who are conducting them. In addition to conducting interviews, we have scanned the literature looking for examples, insights, and guidance for using a DE approach. While our results will be shared in a white paper at the end of the year, as well as at the American Evaluation Association conference in November, we have learned that there are particularly appropriate times to use Developmental Evaluation, including:

• When the right path isn’t clear – there are several ways to go forward
• In environments that are emergent, changing, dynamic 
• When it is difficult to plan or predict all possible outcomes 
• When evaluating systems change and/or community-based change initiatives 
• When evaluating public policy and advocacy efforts 
• When working in diverse cultural contexts 
• When an initiative is being implemented in multiple sites by multiple actors

We are increasingly excited about the value Developmental Evaluation adds to the repertoire of evaluation approaches. While I can’t say what difference it would have made to evaluate The Saturn School of Tomorrow using a DE approach, I do know that given the complexity of today’s social issues, and the need for innovative solutions, Developmental Evaluation offers much promise.

If you have been involved in a Developmental Evaluation, we would love to hear about your experiences!

The American Evaluation Association (AEA) recently passed the organization’s first-ever member supported statement on cultural competence in evaluation, a monumental accomplishment. The Public Statement on Cultural Competence in Evaluation was drafted over a seven-year period, originally commissioned by AEA through the 2005 Building Diversity Initiative (BDI), an initiative designed to build and increase capacity in the field by training and nurturing evaluators of diverse backgrounds. The intended goal of the statement is to serve as a representative of the association’s values and commitment to building diversity, and conducting culturally relevant, reputable, and responsive evaluation. AEA recognizes that it is increasingly important for evaluators not only to understand the basics of evaluation such as methodology, data collection, and dissemination, but also to understand the politics behind programming such as institutional discrimination, which can influence and determine the cultural context, the community, the program, and the evaluation.

AEA’s statement acknowledges that cultural competence does not consist of learning a specific set of skills that is generic and cuts across contexts and situations. Instead, cultural competence is a paradigm shift: “Cultural competence is a stance taken toward culture, not a discrete status or simple mastery of particular knowledge and skills.” (pg. 1, Public Statement on Cultural Competence in Evaluation, AEA, April 2011).

The statement stresses that the culturally competent evaluator recognizes four key aspects:

• That culture imbues all aspect of society from the economic to the political;
• That culture is fluid rather than static;
• That the culturally aware evaluator is always self-aware of what she/he brings to the context including but not limited to biases and privileges; and
• That culture affects all phases of the evaluation and therefore must be considered not only within the methodological design of the evaluation but also in staffing, and ethics.

What would this look like? Well, in keeping with the ethos of the statement, a culturally competent evaluator assessing a drug prevention program designed for adolescents of color might acknowledge the cultural differences within as well as between groups. So too, the evaluator might recognize that these differences might be a result of years of power differentials and stratifications within the community. The evaluator might also recognize her/his own cultural and/or professional privilege and acknowledge that those biases and stereotypes might affect the quality of the evaluation design, data collection methods, and reporting. Finally, in conceding the above mentioned aspects, the evaluator striving for cultural competence would make use of methods and ethical principles that accurately address the cultural context, and employ staff and key informants who can provide a cultural lens by which to view the multitude of effects of the program (while admitting that this lens may be only one of a few). This is but one broad and overarching example of course, but it provides a backdrop for which the statement was written.

The statement’s closing states that cultural competence in evaluation is an on-going conversation with and among one’s colleagues and within one’s self. In striving for cultural competence, evaluators open themselves up to new horizons and broad ways of thinking.  Admittedly the statement is meant to be representative of the times and context during which it was written; thus the expectation is that it is a “living document,” designed to be updated to reflect the cross-sectional times in which it lives.  The statement “went live” on AEA’s website in April 2011 with several web resources added to supplement, and these are available to the public. The resource section is dynamic; evaluators are encouraged to add relevant and helpful readings or materials at any time.

Additional Resources:

Please see the links below for more information concerning AEA’s Public Statement on Cultural Competence in Evaluation.

• Links to Resources on Cultural Competence in Evaluation
• Favorite Citations on Cultural Competence in Evaluation
• Annotated Bibliography: Multiculturalism and Cultural Competence in Evaluation

Increasingly, foundations, corporations, and nonprofits are realizing that in order to solve social problems they need to ground their strategy on reliable and valid data. While philanthropy is shifting towards more strategic giving, the evaluation field is also expanding its toolbox from the traditional model of “post hoc” accountability to more forward-looking approaches. These new tools are producing findings and insights that help social program leaders make more informed choices in real time.

Evidence of these shifts can be seen in the professional literature. For example, the following two publications highlight the link between strategy and evaluation in philanthropy:

• GEO and the Council on Foundations published a report called Evaluation in Philanthropy: Perspectives from the Field which reaffirms the role of evaluation in foundation strategy: “Evaluation …is an essential precursor to effective strategy in philanthropy. It produces the data, information and understanding that enable grantmakers to develop and fine -tune their strategies.”
• A recent publication of the American Evaluation Association’s journal New Directions for Evaluation was entirely dedicated to the link between evaluation and strategy. In the journal, Michael Q. Patton and Patricia A. Patrizi, two well respected evaluators, discuss the new role of evaluation in strategy “Evaluation is a support for strategy. First and foremost, evaluation must be seen and positioned as a key support for strategy development and management; it should have a seat at the strategy table.”

While these examples signal an interesting paradigm shift, nothing beats the inspiration of real-world experience. Through my work at FSG, I have been supporting the work of Battle Creek Unlimited (BCU) (which is heavily funded by the W.K. Kellogg Foundation). BCU is an economic development organization implementing a plan to revitalize downtown Battle Creek, Michigan. The initiative’s primary goal is to establish the city as a national leader in food science and food safety research. FSG has been engaged for more than a year in evaluating this project, which involves a varied group of stakeholders from the Kellogg Foundation, Kellogg Company, community organizations, local businesses, universities, and governmental agencies.

Working at this intersection between strategy and evaluation is very interesting and rewarding. I’m very excited to be part of this movement of evaluators that are using data to help organizations be more strategic in evaluating how they help people and communities in need. I’m curious to learn more about your experience using evaluation to make more informed decisions and I want to invite you to share your experiences with me in the comments section below.

FSG has been thinking about the role of philanthropy in news, information and media for more than two years, as part of our ongoing partnership with the John S. and James L. Knight Foundation – a national philanthropic leader in this area. (Their work is featured in Do More than Give as an example of catalytic philanthropy.) In 2009, FSG was asked to evaluate Knight’s Community Information Challenge, a five-year, $24M initiative that intends to increase engagement of community and place-based philanthropy in supporting local news and media. Knight wanted to catalyze experimentation – get foundations to change their behaviors and utilize digital media (the internet, social and mobile media) to create social change.  And, unlike traditional program evaluation, FSG was asked to emphasize learning – for the Foundation, for grantees, and for the field. Thus began our involvement in a developmental strategic evaluation of the KCIC. (See FSG’s reports in our Knowledge Exchange for more on our findings.)

Our work has involved a near real-time cycle of collecting and reporting data on projects, grantees and the field of community and place-based foundations. Assisted by reports, briefs, memos, slide decks, case studies, and other deliverables we’ve produced, we also talk frequently with our clients and other stakeholders to discuss the evolution of the program and help them think about how to adapt their strategies and approaches. I used to tell people that I wasn’t sure whether we were conducting an evaluation or advising a strategy – and now I am not sure that the difference matters. Our approach to evaluation feels highly appropriate to the dynamic and swiftly changing world of digital media. When the KCIC launched, no one had an iPhone. Twitter and Facebook were fledgling (as one article notes, the “Like” button didn’t exist until April 2010). But today, we need to adapt more quickly – to tablets, foursquare, and whatever the next thing is.

Halfway in, we’ve seen impressive uptake in the field. Roughly 400 community and place-based foundations have applied for matching grants, and 200 foundation CEOs and board members have attended Knight’s annual Media Learning Seminar. To date, Knight has made grants to 84 projects in communities around the United States – in New York City, Los Angeles and Chicago, as well as rural Alaska, Arkansas, Wisconsin and Wyoming. Considering that there are fewer than 800 community foundations in the U.S., a considerable proportion of the field is becoming more aware and incorporating information and media grantmaking into their work.

• They’re using information to further their impact on core community issues, like education, health and the environment.  Some examples include the Community Foundation Serving Boulder County’s Ready. Set. Learn. campaign to increase public funding for  early childhood education, and California Endowment’s Youth Led Media Outlets initiative.
• They’re building the capacity of nonprofits- their grantees – and orienting them to the new digital world so they can better leverage online reach and engagement to create social change.  This may include social media training, which the Northern Chautauqua Community Foundation provided to get more nonprofits to participate in their Amazing County project, or creating a central online hub for nonprofits to better coordinate their efforts to improve the environment, like the Community Foundation for Greater Buffalo did.
• They’re supporting new models of news and journalism. As traditional newspapers have struggled with the old print model, new online models have leveraged the internet, mobile and social media platforms to distribute local independent and public interest news. For example, check out the “upstarts” in Connecticut: the New Haven Independent (funded by the Community Foundation for Greater New Haven) and CT Mirror (funded by the Hartford Foundation for Public Giving).
• They’re building their own capacity to better serve as information providers.  Multiple foundations have hired web and social media staff, for example, to ensure that they are reaching and engaging residents online.

In many ways, foundations are using information and media to further their roles as philanthropic leaders. In particular, they are helping to create actionable knowledge in their communities and providing an opportunity for citizens to take action – whether it’s about decommissioning the local nuclear power plant, voting on a local education funding measure or learning about the perspectives of young people growing up in the community.

Knight and FSG are interested in learning more about what other foundations – in the field, beyond KCIC grantees – are doing to support news and information in their communities. We invite you to tell us your stories and what you’ve learned.

This historic opportunity has come about in part due to ground-breaking research and evaluation conducted over the years.  Studies of the Perry Preschool Project and the Carolina Abecedarian Project have helped establish early learning as one of the most high-impact and cost-effective interventions available to policy-makers.

Philanthropic foundations continue to invest in early learning initiatives and to support the use of data to demonstrate impact and build momentum.  For example, the BirthToFive Policy Alliance, supported by the Buffett Early Childhood Fund, the W. K. Kellogg Foundation, and others provides links to databases, research, and advocacy resources to inform early childhood policy debates.

As government and philanthropy work to create systems that improve outcomes for the youngest and the most vulnerable children, especially low-income and racial minority children, research, evaluation, and data should continue to guide strategies.  

Here are three really great sites that display data on early learning and educational outcomes.

• Kids Count Data Center, by the Annie E. Casey Foundation, catalogs and displays a wealth of early learning indicators, including pre-school enrollment, reading proficiency levels, and children with a family member who reads to them.

• The Scott Foundation for Public Education’s report and site, 50-State Report on Public Education and Black Males 2010, reveals heart-breaking disparities between black and white male high school graduation rates, state by state.

• The United States Education Dashboard, developed by the U.S. Department of Education, includes indicators of 3- and 4-year old preschool enrollment and 4th graders proficient on NAEP reading and math.  (Thanks Matt W!)

To states preparing their applications for the RTTT-ELC: the nation’s future and the promise of social justice are at stake!  Good luck!

On Wednesday, June 15th more than 500 people attended the webinar Focusing on What Matters: Engaging Stakeholders in Developing Evaluation Questions co-hosted by FSG and the Robert Wood Johnson Foundation. The webinar focused on a five-step process for engaging stakeholders, outlined in A Practical Guide for Engaging Stakeholders in Developing Evaluation Questions, which highlights the opportunities and challenges of creating an evaluation process that accommodates different perspectives.

In the end, the conversation about stakeholder engagement went beyond just identifying and prioritizing key questions for the evaluation and into ways of engaging the right stakeholders at the right time in the evaluation process. For example, participants asked,

•  How might you engage grantees in developing a logic model that can be used a basis for developing the evaluation’s key questions?
• How do you provide feedback to stakeholders on which evaluation questions will ultimately be pursued after you have solicited input and feedback from different constituencies?
• When might you want to engage stakeholders individually as opposed to together as a group in order to get their input into developing the key evaluation questions or for other evaluation activities?