AEA365

John LaVelle on Other Evaluation-Related Organizations

Sheila Robinson — Mon, 17 Jun 2013 08:15:20 +0000

My name is John LaVelle, I am the Director of Operations and External Affairs for the School of Behavioral & Organizational Sciences and the School of Politics & Economics at Claremont Graduate University. I’m also a PhD candidate specializing in Evaluation and Applied Research Methods, and a past curator of the AEA365 blog. Today I’m going to share three Rad Resources about some organizations whose work overlaps with Evaluation and also maintain active Listservs as a way of sharing ideas and resources.

Rad Resource: The Society for Community Research and Action. Several posts have been written about the Society for Community Research and Action, which is also known as Division 27 of the American Psychological Association. They are dedicated to advancing theory, research, and social action, and much of their work overlaps with evaluation. SCRA maintains a very active listserv (similar to EvalTalk), and regularly share information about conferences and professional development, research and work opportunities, as well as professional challenges and questions.

Rad Resource: ARNOVA is the Association for Research on Non-Profit Organizations and Voluntary Action. Many evaluators work in a not-for-profit context or with organizations trying to increase volunteer and philanthropy activities, and ARNOVA is a good resource. Similar to AEA’s TIG structure, ARNOVA has areas of specialization, such as community grassroots, social entrepreneurship, and even a section for individuals working at the intersection of practice<—->academics (they call this intersection “pracademics”). ARNOVA’s listserv and webpages have much information about resources, conferences and meetings, calls for papers, etc.

Rad Resource: CBPR stands for Community Based Participatory Research (see Laura Myerchin Skarloff’s excellent post) and the University of Washington hosts a very useful listserv for sharing ideas and problem-solving. They even offer a parallel listserv just for sharing work and research opportunities. Membership information is available here.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Linda Cabral and Laura Sefton on Utilizing Survey Data in a Traditional In-Person Interview

Sheila Robinson — Mon, 17 Jun 2013 01:01:03 +0000

Greetings. We are Linda Cabral and Laura Sefton from the University of Massachusetts Medical School, Center for Health Policy and Research. We are part of a multi-disciplinary team evaluating the Massachusetts Patient Centered Medical Home Initiative (MA-PCMHI), a state-wide, multi-site demonstration project engaging 46 primary care practices in organizational transformation to adopt the PCMH primary care model. To adopt a mixed methods approach, this evaluation utilizes 1) multiple surveys targeted at different stakeholders (e.g., staff, patients), 2) analysis of cost and utilization claims, 3) practice site visits, and 4) interviews with Medical Home Facilitators (MHFs).

We wanted to connect data from the TransforMED’s Medical Home Implementation Quotient (MHIQ) survey with our MHF interview data. We did this to better understand the practices’ MA-PCMHI experience. MHFs provide a range of technical assistance to aid their assigned practices in their transformation process, making them a great source of information about their practices’ transformation. In an effort to triangulate our evaluation findings, we presented the MHIQ results to the MHFs as part of a traditional semi-structured interview. Presenting site specific survey data to MHFs served the following purposes:

It allowed for MHFs to share their reflections on why their practices scored the way they did on various domains;
It prompted MHFs to point out major differences between their assigned sites;
Focused the MHFs on providing practice-specific information; and instead of generalities across all the sites to which they were assigned
MHFs provided insight into some of the strengths and limitations of the survey instrument.

Lessons Learned

Sharing survey data and having respondents reflect on it during the course of an interview, connecting data, proved to be a very helpful strategy. Specifically, we received more detailed responses from interviewees by asking “Why do you think Practice ABC scored a 5 on the care coordination module”? vs. “What can you tell me about how Practice ABC is implementing care coordination?” MHFs would make the case for or against why a practice scored the way they did on a particular domain.
Involving the MHFs as “experts” on their assigned sites increased the MHFs’ investment in the evaluation process and their willingness to participate in future evaluation activities.

Hot Tip

We held these MHF interviews prior to doing practice site visits. The practice-specific information that MHFs shared with us deepened our familiarity with the sites prior to conducting site visits.

Rad Resources

Susan Kistler on Linking in to LinkedIn

Susan Kistler — Sat, 15 Jun 2013 21:37:05 +0000

I am Susan Kistler, the American Evaluation Association’s Executive Director Emeritus and aea365’s regular Saturday contributor.

Rad Resource – American Evaluation Association’s LinkedIn Group: AEA’s LinkedIn group is open to members and nonmembers alike from around the world and currently welcomes over 11,000 colleagues to the community exchange.

Lessons Learned – What is LinkedIn: At its most basic level, LinkedIn is a website dedicated to bringing together those with a common interest into groups for discussion and networking. Joining is free and there are thousands of “groups” within LinkedIn, AEA’s being only one. Once you have set up a free LinkedIn account, you may then join up to fifty groups. Here are just a few that may be of interest to evaluators:

African Evaluation Association (AfrEA) – focusing on colleagues living and working in Africa, this new site hosted by AfrEA just recently came online
American Association of Public Opinion Research (AAPOR) – great discussions in particular for those conducting surveys or in need of sampling guidance
Environmental Evaluators Network (EEN) – our colleagues at EEN use this open group as their primary forum for exchange around environmental evaluation issues
Small and Medium Enterprise (SME) Evaluation – hosted by the World Bank’s Independent Evaluation Group, here you can discuss work with SME’s and also contribute to the group’s study of SME evaluation

Lessons Learned – What types of things are discussed on LinkedIn: LinkedIn discussions are threaded, meaning that a discussion stays together and may be contributed to over the course of many days or even weeks or months. Here are a couple of recent discussions on AEA’s LinkedIn Group (feel free to join in and add to the exchange):

Marcela asked “I am looking for a five point scale to measure improvement. I am currently using: None, Slight, Some, Much, Huge, but my client doesn’t like it. Any suggestions?” and 35 responses offered suggestions, raised further questions, and refined thinking on the topic
Mary Alice asked “Does anyone have any experience assessing changes in non-academic skills (attitude, resiliency, motivation, consciousness, grit, etc) in high school age youth. Preferably, I am looking for an assessment that could be used in a pre/post format…” and 23 posts contributed to the exchange
Katherine asked “Anyone got suggestions for [affordable] stat packages that I might consider as an alternative [to SPSS and SAS] for my own consulting work?” and 24 people shared their expertise

Consider contributing or raising a question!

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Harlan Luxenberg on Creating Streamlined, Visually Appealing Reports with Crystal Reports

Sheila Robinson — Fri, 14 Jun 2013 08:13:30 +0000

Hello! I am Harlan Luxenberg, the Chief Operating Officer of Professional Data Analysts, Inc. (PDA), a small firm in Minneapolis specializing in public health evaluation. Over the years we have worked on multiple projects where we needed to do one or more of the following:

report to a diverse set of stakeholders about program outcomes
report on the comparability of outcomes across multiple projects or summative outcome targets
create report cards or dashboard-like reports for multi-site evaluations

In our search for a powerful software suite that would allow us to efficiently do these tasks while producing visually appealing reports for our clients, we found Crystal Reports. Crystal Reports is not well known in evaluation circles since it is primarily used in the financial industry. But don’t let that fool you! We have been using it over the past eight years to provide reports to educational institutions, health care providers, and individual stakeholders.

(Share Clip)

We have found Crystal Reports to be one of the most useful programs in our data visualization toolkit. While the latest version (2011) does cost nearly $500, you can download a free 30 day demo or buy an earlier version online (like 2008) for less than $400. We have even used Crystal Reports in conjunction with LimeSurvey, an open source and completely free online survey tool. To see more about why we love using LimeSurvey and our experience using it in our evaluations, visit our blog posts on it here.

(Share Clip)

Hot Tip: Create a report template to save time and reduce the potential for errors. To create similar looking reports for different grantees that you are evaluating, simply put the data into a worksheet or database (like Excel). Then connect Crystal to your dataset and you’re ready to create an attractive grantee-specific report that looks similar across grantees. Each report can be developed to only use data from an individual grantee.

Hot Tip: It’s secure! When using fancy dashboard software or even Excel, you often have to give your clients access to your raw data. With Crystal Reports, you can export reports into various formats (like PDF), or your clients can access reports online or through a viewer (both very easy to do). This is especially useful if you are comparing one organization’s data against others and do not want to provide raw aggregate data to everyone.

Resource: If you’d like to see Crystal Reports in action, you can view a template for a standard grantee-specific report that we created for processing Olweus Bullying Prevention Program data.

Lesson Learned: Learning a new software program can be hard! With Crystal Reports, there are extremely helpful online forums where other users will help answer your questions. My favorite is Tek-Tips.

Chris Lysy on Cartoons and Illustrations as Information Visualization

Sheila Robinson — Thu, 13 Jun 2013 08:15:44 +0000

My name is Chris Lysy, I am a Research Analyst at Westat, the creator of Eval Central, and cartoonist at freshspectrum. You can also find me on twitter (@clysy).

At first cartooning was just a hobby, but over time I’ve started to discover the utility of a well designed cartoon. The right cartoons can be potent tools for introducing complex concepts, engaging audiences, and sparking discussion.

On my blog I’ve been experimenting with the fine line between chart junk and good illustration design. A topic briefly addressed by Edward Tufte on his website. Here are some tips I came up with to help you avoid crossing the line.

Hot Tip: 95% Concept, 5% Illustration. If you have a great concept, the illustration doesn’t have to be that great. I spend the vast majority of my time on the concept. Check out the following cartoon from my attribution post. Not the best illustration, but the point is made.

Hot Tip: Be Abrupt. Subtlety thrives in other forms of communication. You have a few seconds to capture attention, just say what you want to say. Here’s an example from my What is Evaluation? post.

Hot Tip: Costello your Abbott. Visuals should always complement the point, not simply repeat the point. In the famous sketch, Costello didn’t completely parrot Abbott, and your visuals should not completely parrot your points. Here is an example from my counterfactual series. My cartoon complements David Henderson’s point. They work together.

“Outliers do not make for compelling client testimonials. Use your metrics to identify what the average experience in your program looks like, and get testimonials from people who fit this profile.”

Hot Tip: Push the Limits. Cartoons give you a certain latitude to go over the top. When used wisely, pushing the limits can spark great discussion. Here is one of my early cartoons from a post on how to not really evaluate.

Angie Ficek on Using the Framing Function in Word to Wrap Text Around Charts

Sheila Robinson — Wed, 12 Jun 2013 08:15:47 +0000

My name is Angie Ficek and I am a Program Evaluator at Professional Data Analysts, Inc., a small firm in Minneapolis, MN specializing in public health evaluation. Last year as I was writing a report, I had the burning desire to wrap a paragraph of text around a smaller chart. This is usually easy to do in Word, except I was using automatic captions for my chart titles so that my figures and tables were automatically numbered. Therefore, my chart titles were outside of my chart and did not travel well with the chart itself.

With a little Google search, I discovered a little known Word function that allows me to create a frame around a chart, its title, and anything else I want grouped together. The framed components can then be moved around as one object.

Hot Tip: I first had to add the framing function to my Quick Access Toolbar in Word. To do this (in Word 2010), go to File > Options > Quick Access Toolbar menu. Make sure that All Commands is selected in the drop-down menu, and then find Insert Frame from the list of commands. Select Add to add this function to your Quick Access Toolbar and click Ok. Now a bright green circle should appear in your Quick Access Toolbar (located along the top left corner of your screen).

When you want two or more components framed together, such as a chart and its title, first highlight the components with your mouse and then click the green Format Frame circle. A frame will now appear around the selected components.

I recommend changing the color of the frame from black to white. To do this, right click on the frame and select Borders and Shading. Then select a white border. Move the frame to adjust its size and place it in the desired location.

Here is an example of a framed chart and caption next to a paragraph of text:

Lessons Learned:

The components that you frame still maintain their original functions. Automatic captions still automatically update. Charts can still be edited.
You can always remove a frame if you decide it’s not needed. Simply right-click on the frame, select Format Frame, and click Remove Frame.
Avoid having one line of text lingering above or below a frame. I always align my frame along the right side of text and then either align it with the first or last line of the paragraph.

Clara Hagens on MEAL in Emergencies Resource Pack

Sheila Robinson — Tue, 11 Jun 2013 08:15:14 +0000

I’m Clara Hagens. I work for Catholic Relief Services (CRS) as the Regional Technical Advisor for Monitoring, Evaluation, Accountability and Learning (MEAL) in Asia. I’d like to share with you a resource pack we have developed to support teams to develop and implement monitoring and evaluation (M&E) systems throughout different phases of an emergency response.

Rad Resource: CRS’ Monitoring, Evaluation, Accountability and Learning in Emergencies: A Resource Pack for Simple and Strong MEAL provides guidance on the key principles of MEAL in emergencies and helps staff to design use-oriented M&E systems that collect just enough information to inform high quality and highly-responsive emergency programming. The Resource Pack clarifies what is different about M&E in an emergency versus a non-emergency setting, namely that in an emergency the M&E system must remain dynamic, that gathering the perspective of the most vulnerable groups is valued over rigorous or heavy data collection methods, that teams are responsible for collecting information on changes context, and that results must be used immediately and often daily for the response to continue to meet the community’s needs.

The Resource Pack demonstrates how both information needs and the appropriate mix of methods and respondents to meet those information needs evolves during an emergency response. Please see the figure below for a graphic depicting this evolution.

(Reprinted by permission from Dominique Morel and Clara Hagens, Monitoring, Evaluation, Accountability and Learning in Emergencies: A Resource Pack for Simple and Strong MEAL (Baltimore, MD: Catholic Relief Services, 2012), [page 14].)

The Resource Pack also emphasizes the importance of monitoring community satisfaction with the response and developing feedback mechanisms to increase accountability. Additional topics in the Resource Pack include sampling during data collection (both when and how to sample), conducting a daily debrief session, and selecting an appropriate mix of learning events at different points in the response.

Kathy Muhr, Aniko Laszlo and Alexis Henry on Using Concept Mapping to Evaluate Employment Collaboratives for People with Disabilities.

Sheila Robinson — Mon, 10 Jun 2013 08:15:16 +0000

Hi. We are Kathy Muhr, Aniko Laszlo and Alexis Henry from UMass Medical School’s Center for Health Policy and Research. Through Work Without Limits, a Massachusetts network of employers and providers that aims to increase employment for people with disabilities, we evaluate and promote programs, policies, and practices related to recruitment, retention, accommodation, and career advancement of people with disabilities. One of our efforts has been the development of Regional Employment Collaboratives (RECs), which bring together cross-disability employment service providers to more effectively engage employers and identify ways of producing better employment outcomes.

We conducted a process evaluation of the RECs, asking the question “what does it take to build and sustain a collaborative of disability employment service providers?” using a concept mapping approach. Concept mapping is a participatory, multi-stage, mixed-method approach that, among its many uses, involves stakeholders in describing how programs are developed and implemented. To conduct the evaluation, we invited REC members to participate in “brainstorming” sessions during which they generated statements in response to the above question. Next, members sorted and categorized the statements into similar groups. We then entered the sorted statements into a concept mapping software program, which uses multidimensional scaling and hierarchical cluster analysis to generate a visual representation – a concept map – of how the statements were grouped by the participating members. As a final step, we reviewed our findings with the REC Project Directors, who helped us interpret the data, determine a final number of clusters, and generate meaningful labels that captured the concepts the clusters represented.

The concept map shows the relationship of clusters to each other – clusters closer together on the map represent concepts that are more closely related and those further apart represent concepts less closely related. This approach revealed concepts that were related to the necessary ingredients for building collaboratives in general, as well as concepts that were specific to building collaboratives focused on enhancing employment for people with disabilities. We feel that the concept mapping approach was very effective in getting the first-hand perspectives of the stakeholders involved in building the RECs, and provided us with some strategies for further development and replication of the REC model.

Lesson Learned: Mind the learning curve.Make sure you allow enough time to learn your concept mapping software, and to complete all stages in the concept mapping process.

Lesson Learned: Beware of concept mapping software overload. The Internet provides an extensive list of concept mapping software; some are free and others are not. It is important to select the software that best meets the needs of your project.

Rad Resources: Examples of concept maps and various concept mapping software products.

(Share Clip)

Linda Meyer on Evaluation with English Language Learners (ELLs)

Sheila Robinson — Sun, 09 Jun 2013 08:15:49 +0000

My name is Linda Meyer and I am an applied developmental psychology graduate student at Claremont Graduate University. I am going to provide you with a few basic tips to keep in mind as you are planning an educational evaluation with English Language Learners (ELLs) in PK-12.

Hot Tip: Find out how students were labeled as English Learners. Most students get labeled through a similar process: a home survey sent to parents upon enrollment, followed by a standardized test on English language development, and a score on that test that falls below what has been designated “proficient.” However, policies are often applied inconsistently and inappropriately. These tests may be inappropriate for young students and the school’s criteria may over- or under-identify English Learners. These students then receive inappropriate instruction that negatively impact their academic performance and development. Lawsuits against various states (see Arizona) demonstrate the widespread problems that stem from inappropriate identification and instruction. Before you evaluate the quality of instruction, make sure students are correctly classified and that the instruction matches students’ abilities.

Hot Tip: Consider English Learners’ ages and language exposure. Children at the elementary level are more likely to develop proficiency in a second language than older students because language acquisition ability trails off at about age 12. Students who fail to sufficiently develop English as a second language after a period of years are designated Long-term English Learners and are at higher risk of dropping out of school, among other things. Students who are not exposed often enough to their second language are less likely to become proficient in that language. Consider how often students hear and use English each day. Higher exposure at earlier ages increases the likelihood that they will become English proficient.

Hot Tip: Match your methods to the instruction type. Some instruction types emphasize learning English as quickly as possible, sometimes at the cost of grade level content instruction time. Others emphasize keeping up with the grade level content in whichever language is more proficient, with some moving students toward English-only instruction over time and others moving students toward proficiency in both languages for all content. Each has its strengths and weaknesses, so match your evaluation questions and methods to the instruction type to obtain the most relevant data.

Rad Resource: A comprehensive national study on English Learners’ academic achievement that investigated various instruction types.

Susan Kistler on Exploring Your Evaluand’s History With the Wayback Machine

Susan Kistler — Sat, 08 Jun 2013 20:27:12 +0000

My name is Susan Kistler and I am the American Evaluation Association’s Executive Director Emeritus and aea365’s regular Saturday contributor.

Last week, I saw the new Star Trek movie (yes, my inner geek shines through once again). Part of its premise was that a team had been asleep for years, only to awake and wreak havoc on the Star Trek crew. You would never do that, you would draw on the past to improve the future of course because evaluators are a good lot. That is why today I’m sharing information about the Wayback Machine.

Rad Resource – The Wayback Machine: The Wayback Machine allows you to look at the history of a website over time. You visit the Wayback webpage, enter the URL for the site in which you are interested, and you can see how the site looked historically, with most of the formatting and the links intact.

Clipped from: archive.org (share this clip)

Here’s a screenshot of the AEA website in February of this year:

And in August of 2004:

And in December of 1998:

If you are were exploring it on the Wayback Machine website, you’d be able to try out most of the links and read the content of each page.

Why is it useful? This is a great way to do background research on your evaluand, to understand issues of explore change over time as reflected in their web presence. If your evaluation includes an examination of your evaluand’s communications, the Wayback machine can also help you see how those communications played out online.

There are a few caveats – formatting is better for some sites than for others, and you can’t see a snapshot of every day in history for a site, only at times when the web crawler used by the Internet Archive visited the site.

The Wayback Machine takes all of 2 minutes to use and explore. And, its even a bit of fun, in particular if you are an historian at heart.