<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" media="screen" href="/~d/styles/rss2full.xsl"?><?xml-stylesheet type="text/css" media="screen" href="http://feeds.feedburner.com/~d/styles/itemcontent.css"?><rss xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:wfw="http://wellformedweb.org/CommentAPI/" xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:sy="http://purl.org/rss/1.0/modules/syndication/" xmlns:slash="http://purl.org/rss/1.0/modules/slash/" xmlns:feedburner="http://rssnamespace.org/feedburner/ext/1.0" version="2.0">

<channel>
	<title>Usable Knowledge</title>
	
	<link>http://www.usablellc.net</link>
	<description>Program evaluation and consulting for non-profit organizations</description>
	<lastBuildDate>Thu, 27 Oct 2011 22:07:45 +0000</lastBuildDate>
	<language>en</language>
	<sy:updatePeriod>hourly</sy:updatePeriod>
	<sy:updateFrequency>1</sy:updateFrequency>
	<generator>http://wordpress.org/?v=3.2.1</generator>
<xhtml:meta xmlns:xhtml="http://www.w3.org/1999/xhtml" name="robots" content="noindex" />
		<atom10:link xmlns:atom10="http://www.w3.org/2005/Atom" rel="self" type="application/rss+xml" href="http://feeds.feedburner.com/UsableKnowledge" /><feedburner:info uri="usableknowledge" /><atom10:link xmlns:atom10="http://www.w3.org/2005/Atom" rel="hub" href="http://pubsubhubbub.appspot.com/" /><feedburner:browserFriendly></feedburner:browserFriendly><item>
		<title>Questioning the External Focus of Outcome Evaluation</title>
		<link>http://www.usablellc.net/questioning-the-value-of-outcome-evaluation</link>
		<comments>http://www.usablellc.net/questioning-the-value-of-outcome-evaluation#comments</comments>
		<pubDate>Thu, 27 Oct 2011 13:23:37 +0000</pubDate>
		<dc:creator>Eric Graig</dc:creator>
				<category><![CDATA[Ideas]]></category>
		<category><![CDATA[Strategies]]></category>
		<category><![CDATA[evaluation strategy]]></category>
		<category><![CDATA[outcome evaluation]]></category>

		<guid isPermaLink="false">http://www.usablellc.net/?p=627</guid>
		<description><![CDATA[We came across an opinion piece in  in the Chronicle of Philanthropy by Kelly Campbell and Matt Forti of the Bridgespan Group in which they make a number of arguments for conducting rigorous outcome evaluations of nonprofit programs. While the piece concludes with a statement about the value of ongoing assessment in the service of continuous program [...]]]></description>
			<content:encoded><![CDATA[<p></p><p>We came across an <a href="http://philanthropy.com/article/Let-Evaluation-Evolve/129393/">opinion piece in  in the Chronicle of Philanthropy</a> by Kelly Campbell and Matt Forti of the Bridgespan Group in which they make a number of arguments for conducting rigorous outcome evaluations of nonprofit programs. While the piece concludes with a statement about the value of ongoing assessment in the service of continuous program improvement, our sense was the it mainly emphasizes the external value of evaluation. Often, the goal of such outwardly focused work is the production of evidence about program outcomes; either to satisfy existing funders or to interest new ones.</p>
<p>We believe that it&#8217;s time fornonprofit organizations to push back against such a conception of outcome evaluation. First, of course, evaluation is difficult and expensive to do and few funders are willing to pay for it. Good evaluation&#8211; the kind that really documents the efficacy of a program model&#8211; is very expensive. Second, without a broad-based, large N study, preferably involving randomized control groups, outcomes are often inconclusive. Lastly, most programs, at best, have only very modest effects.  The people nonprofit organizations serve&#8211; at risk youth, substance using adults, the homeless, etc.&#8211; live in devastating situations that may have been a lifetime in the making.  The challenges they face are deeply woven into the fabric of society as a whole, that is to say they are structural, and a single program, even a very well-run and comprehensive program, is not likely to make a substantial and measurable difference for most of its participants.</p>
<p><span id="more-627"></span><br />
What this means is that single case (or single organization) outcome evaluations, done with limited time and dollar resources, are not likely to show much change.  Is this a reason to abandon evaluation? Of course not.  But it does provide a rationale for shifting questions away from those that ask whether an organization has achieved a set of specific outcomes, and towards those that ask, in a more open-ended fashion, what it has acheived, for whom, and under what circumstances.  Those are the important things to look at.  People, communities and indeed whole societies do change, but incrementally and sometimes, only imperceptibly. Effective organizations use evaluation not to justify their work&#8211; does a dance program for inner city kids really need to be justified&#8211; but to deepen their understanding of what they do and how it affects those they serve.  Doing so shows that they&#8217;re in it for the long haul and that in and of itself, ought to interest funders enough to invest.</p>
]]></content:encoded>
			<wfw:commentRss>http://www.usablellc.net/questioning-the-value-of-outcome-evaluation/feed</wfw:commentRss>
		<slash:comments>2</slash:comments>
		</item>
		<item>
		<title>An Alternative to the Evaluation RFP Process</title>
		<link>http://www.usablellc.net/an-alternative-to-the-evaluation-rfp-process</link>
		<comments>http://www.usablellc.net/an-alternative-to-the-evaluation-rfp-process#comments</comments>
		<pubDate>Thu, 06 Oct 2011 17:38:31 +0000</pubDate>
		<dc:creator>Eric Graig</dc:creator>
				<category><![CDATA[Ideas]]></category>
		<category><![CDATA[Strategies]]></category>
		<category><![CDATA[evaluation strategy]]></category>

		<guid isPermaLink="false">http://www.usablellc.net/?p=578</guid>
		<description><![CDATA[This post is the last in our series about hiring an evaluation firm. Just to recap, we&#8217;ve suggested in our previous posts that while technical expertise is absolutely critical to a successful evaluation, soft skills, including flexibility and a strong client services orientation can make or break a project. Many organizations seeking a consultant issue a formal RFP, or [...]]]></description>
			<content:encoded><![CDATA[<p></p><p>This post is the last in our series about hiring an evaluation firm. Just to recap, we&#8217;ve suggested in our previous posts that while technical expertise is absolutely critical to a successful evaluation, soft skills, including flexibility and a strong client services orientation can make or break a project. Many organizations seeking a consultant issue a formal RFP, or request for proposals, believing that it creates a level playing field on which firms can pitch their approaches and compete for business. Typically, the RFP describes the program in question, suggests some high level evaluation questions, and specifies the form the response should take, and the criteria upon which responses will be evaluated. It may offer a budget or budget range. It is important to recognize however that not all evaluation projects begin with an RFP.  Indeed there are good reasons to not employ an RFP process.  What are they?</p>
<p><span id="more-578"></span></p>
<p>First, many firms, particularly smaller ones, often do not have the resources necessary to put together proposals on spec.  Credible responses require a considerable effort to create and the best small firms may be too busy managing existing projects to respond. More important however, even the most well written RFP response will do little to help you evaluate a consultant based on the selection criteria we mentioned earlier. Certainly you&#8217;ll be able to gauge responders’ writing ability and perhaps get an idea of their technical competence, but you won&#8217;t learn much about the soft skills that are so essential to making an evaluation project a success. You won’t learn much, for example, about an evaluator’s flexibility, his or her ability to problem solve or willingness to go the extra mile necessary to make your project a success.  Yet as we have said, organizations get the most value from an evaluation when it becomes a collaborative endeavor involving give-and-take with a trusted advisor.</p>
<p>For these reasons, we recommend you consider setting aside a formal RFP model and select your evaluator more holistically. One approach involves a letter of interest process. It differs from the RFP model in that it asks prospective consultants to share their perspectives on your project— their ideas and approaches— while de-emphasizing the kind of detailed evaluation plan that often changes anyway as projects move forward.  Beyond asking for an overview of their approach however, ask responders to describe the challenges they anticipate and their strategies for dealing with them. Ask them to draft a list of questions for your team. Their responses will be a good indicator of their flexibility and thoughtfulness. In reviewing the letters of intent you receive, first ask yourself whether the firms’ approach aligns with the project as you see it. Are you, in other words, on the same page? Assuming that the firm’s history suggests the technical competence necessary to do a credible job, the key question is this&#8211; do you get the sense that working with them will add value to your organization and help it advance its strategic goals? Do you feel they will make effective thinking partners? Would you enjoy working with them? Again, while it is critical that you verify their methodological competence, these kinds of soft skills are essential to getting the most value out of an evaluation project.  Jane Davidson of Genuine Evaluation has penned a more pointed and somewhat irreverent take on this approach which you&#8217;ll find <a href="http://genuineevaluation.com/9-golden-rules-for-commissioning-a-waste-of-money-evaluation/">here</a>.</p>
<p>Whether you choose to try the letter of intent approach outlined above, or decide to go the route of a traditional RFP, make sure your solicitation includes the following items.</p>

<table id="wp-table-reloaded-id-3-no-1" class="wp-table-reloaded wp-table-reloaded-id-3">
<tbody>
	<tr class="row-1 odd">
		<td class="column-1">A brief history of your organization and of the program in question</td><td class="column-2">This should be no more than a single paragraph especially if your organizations website provides a good background about your work.</td>
	</tr>
	<tr class="row-2 even">
		<td class="column-1">The goals of the evaluation work</td><td class="column-2">See Part I above. If you have started to develop evaluation questions you should include them here.</td>
	</tr>
	<tr class="row-3 odd">
		<td class="column-1">A detailed outline of the program</td><td class="column-2">What are the programs main components? How far are you in terms of implementation? Who are the key participants? How many people are there in each participant group? Where are they located?</td>
	</tr>
	<tr class="row-4 even">
		<td class="column-1">A budget range</td><td class="column-2">This is key.  A given program can be evaluated in many different ways.  Firms responding to your solicitation need to get some idea of the scope of work you had in mind.</td>
	</tr>
	<tr class="row-5 odd">
		<td class="column-1">Expectations (if any) about the role internal staff will play in the work</td><td class="column-2">Will staff assist with data collection? To what degree? Will they themselves be interviewed, etc.?</td>
	</tr>
	<tr class="row-6 even">
		<td class="column-1">Expectations about reporting</td><td class="column-2">Are you expecting a formal written report or will a series of shorter summary reports be satisfactory?</td>
	</tr>
	<tr class="row-7 odd">
		<td class="column-1">Due dates and contact information</td><td class="column-2">Due dates are obvious enough but responders may also wish to contact you with follow-up questions about the project. Their questions may be administrative (clarifications about your process) or substantive (questions about the program design). You may choose to delegate these two types of inquiries to different people on you staff.</td>
	</tr>
</tbody>
</table>

<p>You should give responders at least three to four weeks to prepare their proposals, less if you are contacting firms you already know about for whom you have solid email addresses or telephone numbers, and more if the proposal is being released publically.  It generally takes at least a few days for publically released solicitations to get noticed by potential responders.</p>
<p>We hope you have found this series useful and more important that it has given you some new perspectives on how to locate and select an appropriate firm for your next evaluation project. If you have any thoughts you&#8217;d like to share on our ideas,  please don’t hesitate to leave us a comment.  We&#8217;ll try our best to respond right away.</p>
]]></content:encoded>
			<wfw:commentRss>http://www.usablellc.net/an-alternative-to-the-evaluation-rfp-process/feed</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>Hiring an Evaluation Firm- Part 3</title>
		<link>http://www.usablellc.net/hiring-an-evaluation-firm-part-3</link>
		<comments>http://www.usablellc.net/hiring-an-evaluation-firm-part-3#comments</comments>
		<pubDate>Thu, 06 Oct 2011 16:16:53 +0000</pubDate>
		<dc:creator>Eric Graig</dc:creator>
				<category><![CDATA[Ideas]]></category>
		<category><![CDATA[Strategies]]></category>
		<category><![CDATA[evaluation strategy]]></category>

		<guid isPermaLink="false">http://www.usablellc.net/?p=563</guid>
		<description><![CDATA[Surfacing an Evaluation Firm In the last post we discussed seven characteristics to look for in an evaluation consultant. In this post, we’ll discuss how to actually go about surfacing one and how to manage the hiring process. In other words we&#8217;ll look at how to narrow your choices down to a short list of [...]]]></description>
			<content:encoded><![CDATA[<p></p><h4>Surfacing an Evaluation Firm</h4>
<p>In the last post we discussed seven characteristics to look for in an evaluation consultant. In this post, we’ll discuss how to actually go about surfacing one and how to manage the hiring process. In other words we&#8217;ll look at how to narrow your choices down to a short list of consultants from which to choose. First, of course—how do you locate prospective consultants?  It probably doesn&#8217;t come as a surprise for us to suggest that colleagues in your field are probably the best place to start.  But when you’re given a name, don’t take it at face value.  Ask about your colleague’s experience with the consultant keeping in mind the list of desirable characteristics we mentioned earlier.</p>
<p>But what do you do if you don’t have a colleague who has worked recently with an evaluator.  Where else can you look? First try asking your funders.  They may know of a firm based on their work with other nonprofits or through their own experience with evaluation firms.  A local United Way may be appropriate as well. Another great source may be a local, regional or national association of organizations in your field.</p>
<p><span id="more-563"></span></p>
<p>You’ll probably want to review four or five proposals before you create a short list of firms to interview.  If the referral sources mentioned above don’t generate a sufficient number of leads, you may need to consult a directory. Probably the best source in the evaluation space is the online directory maintained by the American Evaluation Association.  They list individual evaluators and evaluation firms in all fifty states.  The significant advantage they provide is some level of certainty that the individuals on their list are actually evaluation professionals—you have to be a member of the AEA to have a listing.  This may not be the case elsewhere. Indeed some directories allow consultants to list so many practice areas, surfacing a real specialist can be next to impossible.  If you’ve read our previous post, you know that at least as far as evaluation is concerned, technical expertise is critical to a successful project. Consultants who bill themselves as generalists may not have it.</p>
<p>Here is a partial listing of some consultant directories you may want to take a look at:</p>

<table id="wp-table-reloaded-id-2-no-1" class="wp-table-reloaded wp-table-reloaded-id-2">
<tbody class="row-hover">
	<tr class="row-1 odd">
		<td class="column-1">American Evaluation Association</td><td class="column-2"><a href="http://www.eval.org">www.eval.org</a></td><td class="column-3">Probably the best place to start.</td>
	</tr>
	<tr class="row-2 even">
		<td class="column-1">Nonprofit Central</td><td class="column-2"><a href="http://www.npocentral.net/index.php?l=1">www.npocentral.net/index.php?l=1</a></td><td class="column-3">To be listed with this site consultants need to submit a recommendation which is verified BEFORE the listing goes live.</td>
	</tr>
	<tr class="row-3 odd">
		<td class="column-1">Southern New England Nonprofit Consultant Directory</td><td class="column-2"><a href="http://www.sneconsultant.org/search.asp">http://www.sneconsultant.org/search.asp</a></td><td class="column-3">Covers southern New England. Some great resources on the selection and hiring process.</td>
	</tr>
	<tr class="row-4 even">
		<td class="column-1">Idealist.org</td><td class="column-2"><a href="http://www.idealist.org/"www.idealist.org>www.idealist.org</a></td><td class="column-3">Idealist doesn’t list consultants per se but many of the organization in their database are consultancies.</td>
	</tr>
</tbody>
</table>

<p>Many organizations seeking a consultant issue a formal RFP, or request for proposals, believing that it creates a level playing field on which firms can pitch their approaches and compete for business. It is important to recognize however that not all evaluation projects begin with an RFP. In our next post, we&#8217;ll cover some alternatives to the formal RFP.  Stay tuned!</p>
]]></content:encoded>
			<wfw:commentRss>http://www.usablellc.net/hiring-an-evaluation-firm-part-3/feed</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>Hiring an Evaluation Consultant- Part 2</title>
		<link>http://www.usablellc.net/hiring-and-evaluation-consultant-part-2</link>
		<comments>http://www.usablellc.net/hiring-and-evaluation-consultant-part-2#comments</comments>
		<pubDate>Mon, 12 Sep 2011 23:37:28 +0000</pubDate>
		<dc:creator>Eric Graig</dc:creator>
				<category><![CDATA[Ideas]]></category>
		<category><![CDATA[Strategies]]></category>
		<category><![CDATA[evaluation strategies]]></category>

		<guid isPermaLink="false">http://www.usablellc.net/?p=550</guid>
		<description><![CDATA[In the last post we discussed five different types of evaluation projects.  Here, we’d like to describe seven key characteristics of an effective evaluation consultant. For each one, we’ve provided some insights into how to judge whether the consultants you are looking at possess the qualities you’re looking for— how to evaluate your evaluator in [...]]]></description>
			<content:encoded><![CDATA[<p></p><div>In the last post we discussed five different types of evaluation projects.  Here, we’d like to describe seven key characteristics of an effective evaluation consultant. For each one, we’ve provided some insights into how to judge whether the consultants you are looking at possess the qualities you’re looking for— how to evaluate your evaluator in other words.  A future post will offer some tips for the hiring process itself. Here are the seven skills:</div>
<div><span style="color: #ffffff;">.</span></div>
<div>
<ul>
<li>Technical research expertise</li>
<li>Project management skills</li>
<li>Excellent thinking skills</li>
<li>Flexibility</li>
<li>An orientation towards collaboration</li>
<li>Excellent writing and communication skills</li>
<li>A strong client services orientation</li>
</ul>
</div>
<p>The first three attributes have to do with getting your project done.  The next four relate to your consultant’s ability to work with you so that the evaluation has maximum impact and comes off smoothly.</p>
<p><span id="more-550"></span></p>
<p><strong>Technical research skills</strong> are of course paramount.  If you are going to use evaluation to improve your program or simply communicate what you have achieved to your stakeholders, you need to be certain that the findings you report are accurate and complete.  You need confidence, in other words, that data collection and analysis were handled correctly; that your surveys asked valid questions; that any interviews were conducted in a neutral and evenhanded way; that the sample that was drawn was of the correct size; that the statistical procedures that were used were the correct ones and that they were performed correctly; and that any qualitative data that was collected was analyzed in a complete and unbiased manner.</p>
<p>We’ve been concerned for a long time that evaluation can sometimes appears easier than it really is.  After all, we’ve all been interviewed for something in our lives and most of us have completed a questionnaire. Given this situation, you may ask yourself whether you really need a professional evaluator or whether it is simply a matter of outsourcing a set of evaluation related tasks to someone outside your organization.  But in order to really answer your organization’s unique questions, you need to be certain that the data upon which those answers are based, represents what is really going on in your program.  Only a firm with well-developed research skills can assure you that it is.</p>
<p><em>How can you assess a firm’s expertise?</em> The best measure of research expertise is specific training in research methodology. And it’s not enough to see a list of advanced degrees among the firm’s principals.  The consultant managing your project is the one who has to have the training.  Look for degrees, the higher the better, in sociology, evaluation research, psychology and anthropology.  Look for experience teaching research in a university setting as well.</p>
<p>One additional note: While most professional programs require a research course or two, few can match the level of training in methodology students receive in the fields mentioned above.  That said, there are several excellent post-Master’s certificate programs (<a href="http://tei.gwu.edu/certificate.htm">TEI</a>, <a href="http://www.cgu.edu/pages/670.asp">Claremont</a>) designed to give consultants deeper exposure to research.  If you can find an individual who has completed one of these you are most likely in very good hands.</p>
<p><strong>Project management skills</strong> are important too. Your project actually has to get done&#8211; on time and on budget.  To succeed here you need a consultant who understand what it takes to get a survey out and follow-up with non-respondents. You need a consultant who can find a venue for your focus group or track down missing participants from a longitudinal study.  Without these skills, the best laid plans will never be realized.</p>
<p><em>How can you assess a firm’s project management skills? </em>We think the best way is to look at their experience. How long have they been in business? How many similar surveys, interviews, focus groups, have they done in the past? Firms that have lots of experience under their belts usually know how to make a study happen.  Those are the firms you will want to look at the most closely.</p>
<p><strong>Excellent thinking skills</strong> are critical as well.  By excellent thinking skills we mean the ability to interpret data in meaningful ways&#8211; ways that can offer insights and provoke the kinds of questions that help programs and organizations improve what they do. But it goes beyond interpretive ability. A consultant with excellent thinking skills will understand the contexts and settings within which your organization operates and will be able to work with you, as a thinking partner, in ways that go beyond the findings of a single study.  Ultimately, you should be able to rely on such a consultant as a trusted strategic advisor.</p>
<p><em>How can you assess a consultant’s thinking skills? </em>This one is actually a bit more straightforward than you would think.  Education of course plays a role but experience is most important, but it really comes down to their level thoughtfulness.  Do they ask meaningful questions? Can they think on their feet? Do you sense that your prospective consultant can see the forest for the trees, the big picture in other words? Try asking each firm you interview who their favorite evaluation scholar is.  If they can’t answer that one, if they’re not thoughtful about the field of evaluation, how thoughtful are they likely to be about your organization?</p>
<p><strong>Flexibility </strong>is about recognizing that program evaluation takes place in the real world. Evaluation is often underfunded, done under time pressure, and may involve working with stakeholders who are not fully committed to it.  These realities require an firm that realizes that evaluation is not your organization’s main business. Flexibility means understanding that evaluation work can have tremendous value even if it is not done perfectly.  It means, accepting that as in all things, the perfect should not become the enemy of the good. The key is having the expertise to know where and how to make methodological compromises in order to get the project done. We like Bamberger, Rugh and Mabry’s <em>Real World Evaluation: Working Under Budget, Time, Data and Political Constraints</em>.  It has a lot of practical advice to offer on this subject and you can find it <a href="http://www.realworldevaluation.org/">here</a>.</p>
<p><em>How can you assess a consultant’s flexibility?</em> The answer here is pretty straight forward. First of course you ought to be able to get an idea in your initial interview. Does the consultant talk research techniques with you or does he or she ask about your program? The former can sometimes indicate an over emphasis on academic rigor and a concomitant lack of consideration for the kinds of compromises that have to be made to get work done in the real world. One quality often associated with flexibility is comfort with uncertainty.  Try simply asking the firms you bring in to interview to describe an evaluation gone awry and ask how they handled the situation.  Reference checks are also a great way to assess flexibility.</p>
<p><strong>Collaborative</strong> evaluators undertand that no matter how good a job they do collecting data, you know your program better than they do—what makes it tick, what underlying issues it faces.  Your knowledge about what works and what doesn&#8217;t, your questions and concerns, and your thoughts about how to best manage the evaluation have an important role to play  as the evaluation plan develops and the project moves forward.  When it comes time to develop an interpretation of the study’s findings, the consultant you work with should have a willingness to include you in the analytical process as well.  Such an approach results in richer and deeper answers to the evaluation’s key questions.</p>
<p>While a collaborative approach leads to a better evaluation product, importantly, it also increases the chances that the study’s results will be used.  In a collaborative approach, the ownership of findings is shared. Much more than in a ‘go it alone’ model, both you and your evaluator have a shared stake in the project.  You understand the report you get at the end of the project because you had a voice in shaping it.  Because of such an understanding, you and your team will have more confidence in implementing the study’s recommendations.  Collaboration in other words, makes the report actionable.</p>
<p><em>How do you assess how collaborative a consultant’s is? </em>The best way of course is to ask&#8211; both the consultant and his or her references. Two great questions are: “What kind of role do you anticipate we will play in the evaluation project,” and “how do you see that changing as the project unfolds?” If the consultant just tells you that you’ll be helping to arrange interviews or sending out survey reminders, his or her orientation may not be well geared to collaboration.</p>
<p><strong>Excellent communication skills</strong>  are important because the utilization of evaluation findings depends upon the ability to get them out to program stakeholders in a timely manner and in a format and language they understand.  Reports, summaries, etc. need to present data and other evidence that is accessible and easy to navigate.  For many nonprofits the 30 page evaluation report may not be the best way to go.  Your evaluation consultant needs to be able to understand what kind of communication strategy will work best for your organization.  He or she may also need to be open to creating multiple reports for your different stakeholders.  For example, program managers may be best served with a report that emphasizes how a program was deployed whereas your development group, in order to fulfill funder requirement, may need one focused on outcomes.</p>
<p>Just as important is the timeliness of the information.  Evaluation knowledge makes the greatest contribution to organizational improvement when stakeholders receive it in real time.  This lessens the surprises sometimes found in a final report, breaks the content of such a report up into manageable chunks, and generates excitement among stakeholders. A consultant who expects to submit only a final report may not understand this value.</p>
<p><em>How do you assess a consultant’s communication skills? </em>That’s an easy one, just ask for some writing samples.  When you get them, evaluate them for overall tone and readability.  Can you see your stakeholders understanding what the consultant has written? Can you see them viewing it as reasonable and accurate? It’s very much a matter of fit here. Some organizations may need fairly formal reports written in a very authoritative tone.  Others may do better with something more down-to-earth. If you’d like to learn about best practices for reporting and communications, take a look at <em><a href="http://www.sagepub.com/books/Book225545">Evaluation Strategies for Communicating and Reporting</a>. </em>We use it when we teach evaluation and often refer back to it when we start a new project.</p>
<p><strong>Client oriented </strong>means all of the things we have listed so far and more.  It has to do, essentially, with how easy it is to work with your consultant.  We’ve covered the big things like flexibility and collaborativity, but the little things matter as well, perhaps even more so.  A client oriented consultant is one who returns phone calls and emails promptly; one who is trustworthy and accountable; one who understands confidentiality and who has your best interests at heart.  He or she is willing to go above and beyond the minimum requirements of the work plan or contract and to get the job done for you on time and within budget. <strong></strong></p>
<p><em>How do you know if a consultant is client oriented?  </em>This is really the hardest thing of all to judge. Of course flexibility and collaborativity play some role, but at the end of the day it’s very much a judgment call.  You may get some inkling in the back and forth that typically occurs during the selection process but besides following up with references, you will most likely need to check in with your right brain on this one.  Trust your instincts and if in doubt, move on if the relationship doesn’t feel right.</p>
<p>&nbsp;</p>
<p>In our final post in this series, we’ll cover the process of locating and hiring an appropriate consultant for your project.  Please stay tuned for more.</p>
<p><strong><br />
</strong></p>
]]></content:encoded>
			<wfw:commentRss>http://www.usablellc.net/hiring-and-evaluation-consultant-part-2/feed</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>Hiring an Evaluation Consultant-  Part 1</title>
		<link>http://www.usablellc.net/hiring-and-evaluation-firm-part-1</link>
		<comments>http://www.usablellc.net/hiring-and-evaluation-firm-part-1#comments</comments>
		<pubDate>Fri, 26 Aug 2011 17:03:46 +0000</pubDate>
		<dc:creator>Eric Graig</dc:creator>
				<category><![CDATA[Ideas]]></category>
		<category><![CDATA[Strategies]]></category>
		<category><![CDATA[evaluation strategy]]></category>

		<guid isPermaLink="false">http://www.usablellc.net/?p=523</guid>
		<description><![CDATA[Hiring consultants of any kind is always a challenge for not-for-profit organizations. Unlike a physical product you don&#8217;t know what you&#8217;re really getting until you&#8217;ve signed the contract.  Moreover, consulting services are typically very expensive, highly customized and the results of the work are sometimes critical to your organization&#8217;s success.  In a study we did [...]]]></description>
			<content:encoded><![CDATA[<p></p><p>Hiring consultants of any kind is always a challenge for not-for-profit organizations. Unlike a physical product you don&#8217;t know what you&#8217;re really getting until you&#8217;ve signed the contract.  Moreover, consulting services are typically very expensive, highly customized and the results of the work are sometimes critical to your organization&#8217;s success.  In a study we did for the <a href="http://http://www.unitedwaynyc.org/?gclid=CPrirZHg8qoCFYp25Qod62w-VQ">United Way of New York City</a> (UWNYC) some years ago, we found that while most of the consulting engagements UWNYC funded for member organizations were successful, several wound up doing more harm than good and in fact actually set the client back.  We learned a lot about what works and what doesn&#8217;t work in consulting and what we found can be reduced to four key points:</p>
<ul>
<li>The consultant was difficult to work with (he or she was often inflexible or unavailable or did not get along with staff or only thought in terms of billable hours)</li>
<li>The consultant had a canned, one size fits all &#8216;process&#8217; and was unwilling to collaborate or adequately understand the nonprofit&#8217;s unique situation&#8211; the consultant in other words was a know-it-all</li>
<li>The consultant was unable to understand the nonprofit&#8217;s &#8216;big picture&#8217; or the environment in which it operated</li>
<li>The project failed to build the nonprofit&#8217;s internal capacity or help it grow in a strategic way</li>
</ul>
<p>The study focused on consulting in general and was not specific to program evaluation though we did look at a number of evaluation projects. Since we are sometimes approached by executives in the nonprofit sector with stories of evaluations gone bad, we thought it might be useful to share some thoughts about how to hire an evaluation consultant.</p>
<p><span id="more-523"></span></p>
<p>The place to start is with the goals of the project.  What do you hope to accomplish? Here are five different kinds of evaluation projects we have been involved in along with some thoughts about the kinds of skills an evaluation firm needs to have in order to manage each one:</p>

<table id="wp-table-reloaded-id-1-no-1" class="wp-table-reloaded wp-table-reloaded-id-1">
<thead>
	<tr class="row-1 odd">
		<th class="column-1">Project Goals</th><th class="column-2">Comments</th>
	</tr>
</thead>
<tbody class="row-hover">
	<tr class="row-2 even">
		<td class="column-1">Emergency evaluation</td><td class="column-2">You’re in the middle of a program (or worse, at the end of it) and the funder reminds you that you were supposed to evaluate it.  You need a consultant who is flexible enough to deal with a difficult situation, can work under considerable time pressure and understands that not every project can be completed with the highest standards of academic rigor.</td>
	</tr>
	<tr class="row-3 odd">
		<td class="column-1">Evaluation of new initiative</td><td class="column-2">A new program has been funded and you need an evaluation plan that will help you monitor progress and assess outcomes throughout the life of the program. You need a firm that can think creatively and anticipate the many ways in which evaluation data can be used and one that has the ability to stick with your project over the long haul. Perhaps put best, you need a firm that can partner with you and help make your program a success.</td>
	</tr>
	<tr class="row-4 even">
		<td class="column-1">Evaluation system</td><td class="column-2">You’ve decided that rather than thinking about evaluation on a program by program basis, you would like to incorporate it strategically into your organization’s work. You need a firm that understands that such a system is likely to be something you will need to know how to operate, in other words, one that can coach and train you on how to perform routine evaluation work.  At the same time, your consultant needs to appreciates that incorporating evaluation does not mean making evaluation your organization’s primary goal. Most important of course, you need a firm that has the experience to anticipate the many ways evaluation data can be used to support your organization’s goals.</td>
	</tr>
	<tr class="row-5 odd">
		<td class="column-1">Formative or process evaluation</td><td class="column-2">You need to know how a program is actually working out on the ground.  For example, you have questions about whether the program design is being correctly implemented across various sites or you may want to gather data to use in making mid-course corrections.  For this kind of project, you need an evaluator who can understand the key elements of your program design and has the ability to work collaboratively with the program’s leadership team and with the people actually implementing the initiative.</td>
	</tr>
	<tr class="row-6 even">
		<td class="column-1">Summative or outcomes evaluation</td><td class="column-2">You need to know whether your program was able to achieve its goals. The firm you select will need to be sensitive to both expected outcomes and inadvertent ones. It will also need to know what kinds of group comparisons may be relevant. For example, you will most likely want to know which subgroups in your population benefited the most and which benefited the least.</td>
	</tr>
</tbody>
</table>

<p>Of course a competent firm ought to be able to handle any of these kinds of projects—there are no firms that we know of that specialize in one or the other.  Our purpose in laying them out is to encourage you to think about your goals and about the elements of success that are unique to them.</p>
<p>In a future post, we’ll touch on five things to look for as you begin the process of locating and engaging an evaluation firm for your organization.</p>
<p>&nbsp;</p>
]]></content:encoded>
			<wfw:commentRss>http://www.usablellc.net/hiring-and-evaluation-firm-part-1/feed</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>More on Data Visualization</title>
		<link>http://www.usablellc.net/more-on-data-visualization</link>
		<comments>http://www.usablellc.net/more-on-data-visualization#comments</comments>
		<pubDate>Sun, 03 Apr 2011 14:34:27 +0000</pubDate>
		<dc:creator>Eric Graig</dc:creator>
				<category><![CDATA[Uncategorized]]></category>
		<category><![CDATA[data analysis]]></category>
		<category><![CDATA[data visualization]]></category>
		<category><![CDATA[statistics]]></category>

		<guid isPermaLink="false">http://www.usablellc.net/?p=477</guid>
		<description><![CDATA[David McCandless&#8217; TED talk is very specifically on data visualization and can be found here.  He&#8217;s not quite as engaging as Gosling but his ideas are compelling as are his visualizations.  Both presenters talk about the importance of representing data in context, for example by showing rates.  To make this point, McCandless displays a graphic comparing the total [...]]]></description>
			<content:encoded><![CDATA[<p></p><p>David McCandless&#8217; TED talk is very specifically on data visualization and can be found <a href="http://www.ted.com/talks/david_mccandless_the_beauty_of_data_visualization.html">here</a>.  He&#8217;s not quite as engaging as Gosling but his ideas are compelling as are his visualizations.  Both presenters talk about the importance of representing data in context, for example by showing rates.  To make this point, McCandless displays a graphic comparing the total US military budget with that of other countries. Not surprisingly, the U.S. military budget dwarfs that of much of the rest of the world.  McCandless&#8217; point however is that it would make more sense to look at military spending as a percent of GDP.  Figured this way, US spending looks eminently reasonable, compared say to that of Burundi or Eritrea.  But is it?  In the US, military spending is about 4% of GDP, in Finland it&#8217;s about 1.5%, which country is better off?  I don&#8217;t have the figures but perhaps a better metric would be the value of the property the defense budget is predicting.  That&#8217;s how an insurance company would figure it.   Or maybe a measure of personal well being (they are starting to do <a href="http://www.yesmagazine.org/happiness/beyond-gdp-uk-to-measure-well-being">this </a>in the UK as a mater of public policy) or sense of security?  That would be interesting&#8230; well being as a function of military spending! The point here  though is to show that the denominator of the fraction you use has to be chosen with care.</p>
<p>This idea is echoed in a recent article in the <a href="http://www.nytimes.com/2011/04/03/business/03stream.html?ref=technology">New York Times</a> about data visualization:  “The purpose of visualization,” says Ben Shneiderman, founding director of the Human-Computer Interaction Laboratory at the University of Maryland, “is insight, not pictures.”</p>
<p>So there is more to data visualization than just pretty pictures. In this post, and the last one, we&#8217;ve touched on the story the visualization is designed to tell. But there is also a psychology of visual perception to consider.  The use of lines and colors and areas plays a part in how attractive a visualization appears, but also, it turns out, in our ability to apprehend its message.  For this we have to look elsewhere.  <span id="more-477"></span></p>
<p>In the meantime, here are some links we&#8217;ve found on visualization.  Stay tuned for more.</p>
<p>Here are some more links about visualization:</p>
<p><a href="http://www.coolinfographics.com/links/">http://www.coolinfographics.com/links/</a></p>
<p><a href="http://www.perceptualedge.com/">http://www.perceptualedge.com/</a></p>
<p><a href="http://www.perceptualedge.com/examples.php">http://www.perceptualedge.com/examples.php</a></p>
<p><a href="http://www.informationisbeautiful.net/">http://www.informationisbeautiful.net/</a></p>
<p><a href="http://flowingdata.com/">http://flowingdata.com/</a></p>
<p><a href="http://www.excelcharts.com/blog/data-visualization-hierarchy-of-needs/">http://www.excelcharts.com/blog/data-visualization-hierarchy-of-needs/</a></p>
<p><a href="http://www.visual-literacy.org/periodic_table/periodic_table.html">http://www.visual-literacy.org/periodic_table/periodic_table.html</a></p>
]]></content:encoded>
			<wfw:commentRss>http://www.usablellc.net/more-on-data-visualization/feed</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>The Joy of Statistics</title>
		<link>http://www.usablellc.net/the-joy-of-statistics</link>
		<comments>http://www.usablellc.net/the-joy-of-statistics#comments</comments>
		<pubDate>Thu, 24 Mar 2011 02:32:39 +0000</pubDate>
		<dc:creator>Eric Graig</dc:creator>
				<category><![CDATA[Ideas]]></category>
		<category><![CDATA[Methods]]></category>
		<category><![CDATA[data visualization]]></category>
		<category><![CDATA[quantitative]]></category>
		<category><![CDATA[statistics]]></category>

		<guid isPermaLink="false">http://www.usablellc.net/?p=459</guid>
		<description><![CDATA[We recently came across a wonderful site called gapminder.org that we&#8217;ve just begun exploring.  It&#8217;s the brain child of health policy expert Hans Rosling who explains the &#8216;Joys of Statistics&#8217; as nobody else can.  A frequent speaker at the annual TED conference,   Rosling, has put together an hour long presentation on the basics of statistics, the history [...]]]></description>
			<content:encoded><![CDATA[<p></p><p>We recently came across a wonderful site called <a href="http://www.gapminder.org/videos/the-joy-of-stats/">gapminder.org</a> that we&#8217;ve just begun exploring.  It&#8217;s the brain child of health policy expert Hans Rosling who explains the &#8216;Joys of Statistics&#8217; as nobody else can.  A frequent speaker at the annual <a href="http://www.ted.com/pages/about">TED conference</a>,   Rosling, has put together an hour long presentation on the basics of statistics, the history of discipline and most important, why they are so incredibly important in a world drowning in data.  OK, I&#8217;m a bit of a geek, but if you were ever fascinated by those PBS specials in which Richard Feynman or Carl Sagan strolls through Piccadilly Circus musing on the topics such as parallel universes or ancient Babylonian astronomy (somehow tying them to the goings on in Piccadilly Circus) you&#8217;ll love this program.  If you need to understand how to make sense of data, or want a new perspective on how to use it you should tune in as well.</p>
<p><span id="more-459"></span></p>
<p>Consider the humble &#8216;average&#8217;, in quotes here because there are at last three &#8216;averages&#8217; people talk about, the mean, the median and the mode.  Rosling gives an excellent example of one problem with averages.  The average number of legs people have in Sweden is actually 1.999.  While most people have two legs, some people have only one or none.  No one has more than two.  So?  Most people, the VAST majority in fact are, like the children in Lake Wobegon, nearly all are above average.  From here he goes on to a discussion of the importance of looking therefore at how data are distributed (something <a href="http://www.usablellc.net/outcomes-and-distributions-in-prorgam-evaluation">we&#8217;ve been talking about</a> in this space for some time). But there is more.  At minute 24, he starts on a discussion of statistical graphics which were first used by Florence Nightingale to describe preventable fatalities among British soldiers during the Crimean War.</p>
<p>This is just the tip of iceberg.  There is a ton of excellent content and Rosling&#8217;s enthusiasm for his subject is infectious.  If you have to read (or write) a report that will be filled with quantitative data think about going here first for inspiration.</p>
<p>Here&#8217;s Gosling&#8217;s 2006 TED Talk:</p>
<p><object classid="clsid:d27cdb6e-ae6d-11cf-96b8-444553540000" width="334" height="326" codebase="http://download.macromedia.com/pub/shockwave/cabs/flash/swflash.cab#version=6,0,40,0"><param name="allowFullScreen" value="true" /><param name="allowScriptAccess" value="always" /><param name="wmode" value="transparent" /><param name="bgColor" value="#ffffff" /><param name="flashvars" value="vu=http://video.ted.com/talks/dynamic/HansRosling_2006-medium.flv&amp;su=http://images.ted.com/images/ted/tedindex/embed-posters/HansRosling-2006.embed_thumbnail.jpg&amp;vw=320&amp;vh=240&amp;ap=0&amp;ti=92&amp;introDuration=15330&amp;adDuration=4000&amp;postAdDuration=830&amp;adKeys=talk=hans_rosling_shows_the_best_stats_you_ve_ever_seen;year=2006;theme=presentation_innovation;theme=numbers_at_play;theme=technology_history_and_destiny;theme=unconventional_explanations;theme=rethinking_poverty;theme=women_reshaping_the_world;event=TED2006;&amp;preAdTag=tconf.ted/embed;tile=1;sz=512x288;" /><param name="src" value="http://video.ted.com/assets/player/swf/EmbedPlayer.swf" /><param name="bgcolor" value="#ffffff" /><param name="allowfullscreen" value="true" /><embed type="application/x-shockwave-flash" width="334" height="326" src="http://video.ted.com/assets/player/swf/EmbedPlayer.swf" flashvars="vu=http://video.ted.com/talks/dynamic/HansRosling_2006-medium.flv&amp;su=http://images.ted.com/images/ted/tedindex/embed-posters/HansRosling-2006.embed_thumbnail.jpg&amp;vw=320&amp;vh=240&amp;ap=0&amp;ti=92&amp;introDuration=15330&amp;adDuration=4000&amp;postAdDuration=830&amp;adKeys=talk=hans_rosling_shows_the_best_stats_you_ve_ever_seen;year=2006;theme=presentation_innovation;theme=numbers_at_play;theme=technology_history_and_destiny;theme=unconventional_explanations;theme=rethinking_poverty;theme=women_reshaping_the_world;event=TED2006;&amp;preAdTag=tconf.ted/embed;tile=1;sz=512x288;" bgcolor="#ffffff" wmode="transparent" allowscriptaccess="always" allowfullscreen="true"></embed></object></p>
]]></content:encoded>
			<wfw:commentRss>http://www.usablellc.net/the-joy-of-statistics/feed</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>How to Prove that Your Program Works</title>
		<link>http://www.usablellc.net/how-to-prove-that-your-program-works</link>
		<comments>http://www.usablellc.net/how-to-prove-that-your-program-works#comments</comments>
		<pubDate>Wed, 05 Jan 2011 18:51:00 +0000</pubDate>
		<dc:creator>Eric Graig</dc:creator>
				<category><![CDATA[Ideas]]></category>
		<category><![CDATA[science]]></category>
		<category><![CDATA[theory]]></category>

		<guid isPermaLink="false">http://www.usablellc.net/?p=436</guid>
		<description><![CDATA[Most likely you can&#8217;t.  Don&#8217;t waste your time trying. A rather extreme statement, particularly from an organization that does program evaluation.  Nonetheless, we stand by it.  It&#8217;s hard to get too far into a discussion of the notion of proof without talking at least a bit about Karl Popper&#8217;s philosophy of science and his key [...]]]></description>
			<content:encoded><![CDATA[<p></p><p>Most likely you can&#8217;t.  Don&#8217;t waste your time trying.</p>
<p>A rather extreme statement, particularly from an organization that does program evaluation.  Nonetheless, we stand by it.  It&#8217;s hard to get too far into a discussion of the notion of proof without talking at least a bit about <a href="http://plato.stanford.edu/entries/popper/" target="_self">Karl Popper&#8217;s</a> philosophy of science and his key idea of falsifiability.  For Popper it is always easy to find evidence (legitimate evidence, I will add) to support a general idea (in this case that your program works).  If a single instance can be found where the idea doesn&#8217;t hold up however, the idea will be disproven.  The problem is that in principle, it is impossible to test every possible instance. There is an infinity of them.  We can sometimes try to get around this through the use of inferential statistics, all those t-tests and ANOVAs you remember from your stat course that tell you that 95% of the time, the results you found would not have been due to chance.  But what about that remaining five percent? You&#8217;re out of luck.  For Popper, science, knowledge, only advances as ideas are repeatedly tested by scientists (or evaluators, though the field was in its infancy when Popper wrote<em> The Logic of Scientific Discovery</em>) and remains, in any event, always provisional. A<a href="http://www.newyorker.com/reporting/2010/12/13/101213fa_fact_lehrer"> recent article</a> in the New Yorker in fact shows that a few scientists are beginning to question even this. And then there&#8217;s <a href="http://www.nytimes.com/2011/01/06/science/06esp.html?_r=1&amp;hp">this one </a>from the New York Times.</p>
<p><span id="more-436"></span></p>
<p>So how does policy advance? We were involved recently as the local evaluator on a federal project designed to assess the effectiveness of providing certain kinds of support to low income married couples designed to help them keep their marriages together.  There were a little over a dozen sites scattered throughout the country each of which had a local evaluator. A national evaluation organization was tasked with the job of integrating the findings of the local evaluations and producing a report on the overall efficacy of the idea.  We believe that&#8217;s how funders should conceive of evaluation but they almost never do.</p>
<p>Funders certainly have a legitimate responsibility to fund organizations that do <em>good work </em>(as opposed to those that do<em> good works</em>).  The problem is that they have somehow come to believe that good work and statistically significant outcomes based on randomized control group trials are the same thing.  How this came about would make for a fascinating study.  Such a study aside, the fact is that most well managed social programs have positive outcomes.  But they have them only for some of those they are designed to serve, and then only under very specific circumstances.  Evaluation is useful when it can help nonprofits determine for whom and under what circumstances.</p>
<p>Let&#8217;s put it another way.  Which organization would you rather fund, the one that can document good average results, or the one that recognizes that successes are always provisional, that there is a great deal to learn from occasional failure and that ongoing data collection and assessment is the best path towards continuous improvement?</p>
]]></content:encoded>
			<wfw:commentRss>http://www.usablellc.net/how-to-prove-that-your-program-works/feed</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>Surveys for Nonprofits</title>
		<link>http://www.usablellc.net/surveys-for-nonprofits</link>
		<comments>http://www.usablellc.net/surveys-for-nonprofits#comments</comments>
		<pubDate>Sat, 18 Dec 2010 03:12:02 +0000</pubDate>
		<dc:creator>Eric Graig</dc:creator>
				<category><![CDATA[Methods]]></category>
		<category><![CDATA[surveys]]></category>

		<guid isPermaLink="false">http://www.usablellc.net/?p=420</guid>
		<description><![CDATA[Last Friday we led a workshop at the Support Center for Nonprofit Management on survey design. Every time we offer this particular workshop we come away with an appreciation of how difficult developing a survey can be.  What makes survey design challenging is that there are so many elements to juggle. And, complicating matters, successfully dealing with one design [...]]]></description>
			<content:encoded><![CDATA[<p></p><p>Last Friday we led a workshop at the <a href="http://www.supportcenteronline.org/">Support Center for Nonprofit Management</a> on survey design. Every time we offer this particular workshop we come away with an appreciation of how difficult developing a survey can be.  What makes survey design challenging is that there are so many elements to juggle. And, complicating matters, successfully dealing with one design problem often creates another one. Tools such as <a href="http://www.zoomerang.com/">Zoomerang </a>or <a href="http://www.surveymonkey.com/">Survey Monkey</a> make creating an internet survey look easy but the real question is whether your organization can rely on the results to make policy decisions. Maybe, or maybe not.</p>
<ul>
<li>Maybe if your response rate was high, maybe not if it wasn&#8217;t</li>
<li>Maybe if your nonrespondents were essentially similar to those who did respond</li>
<li>Maybe if your respondents understood the questions, maybe not if they didn&#8217;t</li>
<li>Maybe if they answered questions accurately, maybe not if they didn&#8217;t</li>
<li>Maybe if you got a good range of answers, maybe not if everyone said the same thing</li>
</ul>
<p>You get the idea.</p>
<p>We get loads of questions about how to get the most out of a survey and we&#8217;d like to answer some of them in this blog.  Over the coming weeks, we&#8217;ll use this space to provide some answers.  In the meantime, we&#8217;ve posted the Powerpoint from the Friday session <a href="http://www.usablellc.net/PDF_Library/Slides%20for%20Surveys%20for%20Non%20Profits.pdf">here</a>.  You&#8217;ll find our guide to question design <a href="http://www.usablellc.net/PDF_Library/Question%20Types.pdf">here</a>.  Feel free to download and distribute according to the license agreement on the first page of both documents. And do send us any questions you may have.  We&#8217;ll do our best to respond quickly with an answer.</p>
]]></content:encoded>
			<wfw:commentRss>http://www.usablellc.net/surveys-for-nonprofits/feed</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>Post-Training Surveys</title>
		<link>http://www.usablellc.net/post-training-survey</link>
		<comments>http://www.usablellc.net/post-training-survey#comments</comments>
		<pubDate>Wed, 13 Oct 2010 16:45:14 +0000</pubDate>
		<dc:creator>Eric Graig</dc:creator>
				<category><![CDATA[Methods]]></category>
		<category><![CDATA[surveys]]></category>
		<category><![CDATA[training evaluation]]></category>

		<guid isPermaLink="false">http://www.usablellc.net/?p=388</guid>
		<description><![CDATA[You&#8217;ve seen them. You&#8217;ve completed them. We&#8217;re talking here about those short (hopefully)surveys participants are asked to fill out after attending a training class. We call them &#8220;doughnut surveys&#8221; in part because in addition to asking substantive questions about the trainer and the training content, there are invariably a few questions about whether the venue [...]]]></description>
			<content:encoded><![CDATA[<p></p><p>You&#8217;ve seen them.  You&#8217;ve completed them.  We&#8217;re talking here about those  short (hopefully)surveys participants are asked to fill out after attending a training class.  We call them &#8220;doughnut surveys&#8221; in part because in addition to asking substantive questions about the trainer and the training content, there are invariably a few questions about whether the venue was appropriate and whether the coffee and doughnuts were OK.  The questions asked on these surveys are pretty basic but the answers you get back can be used to tweak program content and help trainers improve their platform skills.</p>
<p>Here are some questions we like to include:<span id="more-388"></span></p>
<p>Using a four point HOW OFTEN scale (All of the Time / Most of the Time / Some of the Time / None or Almost None of the Time) we ask participants often:</p>
<ul>
<li>The material covered was appropriate for someone with my level of knowledge of the subject</li>
<li>The content of the training was applicable to my work or my interests</li>
<li>The material was presented in a clear and logical fashion</li>
<li>There was sufficient time in the session to cover the material that was presented</li>
<li>The instructor was knowledgeable of the subject</li>
<li>The instructor provided good examples</li>
<li>The instructor was respectful of students</li>
<li>The instructor was accessible for questions</li>
</ul>
<p>Of course training is supposed to be about giving people in the information they need to adopt new behaviors or change existing behaviors.  Since they&#8217;ve only just completed the training we can&#8217;t ask participants about its actual impacts of their behavior.  Instead we can ask about how likely it is that they will do things differently. We use a a three point LIKELIHOOD scale for this (Very Likely / Somewhat Likely/ Not Likely).  We like this language better than wordings that involve a  &#8217;Do you plan to&#8230;&#8217; structure because we believe it&#8217;s more concrete.  It&#8217;s harder to say you think you will do something than it is to say you are planning to do something.</p>
<p>A series of AGREE/DISAGREE asking about the degree to which the training sensitized participants about the issues it brought up conclude the survey. Of course we also include the usual demographic items as well as an open-ended item where we ask participants to share their thoughts about the content and overall experience. And we ask whether they liked the doughnuts as well. If you would like to obtain a sample survey, drop us a line.</p>
]]></content:encoded>
			<wfw:commentRss>http://www.usablellc.net/post-training-survey/feed</wfw:commentRss>
		<slash:comments>1</slash:comments>
		</item>
	</channel>
</rss>

