The Argument

“… ‘and what is the use of a book,’ thought Alice, ‘without pictures or conversations?’” I sympathize with Alice because I create pictures to both learn and explain challenging topics that are typically only conveyed by words, and I approach deliverables as conversations with the audience. Evaluation is expanding beyond its academia-bounded roots to balance rigor and professionalism with inclusion, accessibility, and engagement. Combining this philosophical shift with advances in technology that let us create and embed graphics with ease, we have the freedom to bring more than tables and bar charts into our work. We’ve used visualizations to describe models and data, so why not methods?

My premise is simple: By complementing our text with pictures, we make the material more memorable and understandable. When people understand what we’ve done, they’re more likely to trust our findings. If they trust our findings, they’re more likely to use them. And if I’m creating the visualizations for myself anyway, why not offer to share them?

Lesson Learned: Comprehension Builds Trust

People want us to be experts, to trust that we know what we’re doing and will deliver credible findings. Operational transparency fosters trust but arguably, that transparency is more effective when they understand what we’re doing. The audience doesn’t need a two-hour masterclass on the intricacies of the chosen method, but adding pictures to methods can increase clarity—and my clients and teammates have expressed appreciation for graphics in presentations and reports. The benefits are twofold: we’re demonstrating that we understand the method well enough to explain it simply, and they get some grounding in the methodology that informs their faith in our findings.

Hot Tips:

• Design principles say to keep it simple—you want the audience to understand, not be distracted or overwhelmed—so no 3-D graphics with shadows and gradients.
• Use the client’s color palette to make the graphic feel custom. PowerPoint, Canva, and Adobe Fresco are great tools for creating images by hand. I love Canva’s AI-assisted tools, like Background Remover, but haven’t had any luck using AI to actually generate usable images yet.

Rad Resource:

• Universal Design principles provide a framework to make your visualizations accessible, understandable, and usable to as great a diversity of people as possible.

Lesson Learned: Your Mileage May Vary

Each reporting context is unique, with some supervisors, teammates, and clients being more receptive to including graphics than others. I do consider myself lucky to have landed in teams that let me push back on the status quo and develop my own way of working. The reverse has also been true, whether due to supervisor or client preferences. I’ve also learned that some people do actually want and need text, finding pictures irrelevant or even problematic. If you think that graphics will be useful, then it’s worth a conversation, but be prepared for a “no.”

Some may contend that adding visuals, especially those with bright colors, will audiences to take us less seriously. I’d argue that there are more Alices out there than we realize.

Do you have questions, concerns, kudos, or content to extend this AEA365 contribution? Please add them in the comments section for this post on the AEA365 webpage so that we may enrich our community of practice. Would you like to submit an AEA365 Tip? Please do so using the form on our website. AEA365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

In the Canadian federal government, policy dictates that evaluations of grant and contributions programs must occur every five years and be conducted in a neutral manner. While working at the Natural Sciences and Engineering Research Council, a federal agency that funds research and innovation, the evaluation division was approached by Ed Irving and Jeffrey Nerenberg, the directors of the newly implemented Alliance Grants, who wanted to understand why participation was lower than anticipated. They had theories but wanted evidence to confirm their thinking and support potential program changes. Although the formal, policy-required evaluation was still two years away, they needed answers now, and it was our responsibility to help them.

When Timing Is Everything

To meet the program’s needs, my director, Susan Morris, suggested a rapid program review to generate quality evidence quickly for learning and decision-making. The review would be narrowly focused and completed in six months.

Rather than producing a formal written report with recommendations, we summarised our findings and opportunities for improvement in a presentation deck. This made the information easier to digest and faster to share. The program could also repurpose the deck for senior management, and other interested parties.

Hot Tip

• If a rapid review is conducted outside the formal evaluation cycle, plan from the outset for how the findings will feed into the eventual evaluation. This keeps the work rigorous and defensible.

A More Participatory Approach for Useful Findings

My colleagues, Kelly-Lyn Christie and Sara Grondin, and I worked closely with the program to identify what they needed to know, maintaining ongoing dialogue about emerging findings and ideas for change.

We found opportunities to join consultations the program was already planning with the research community. This was a novel approach for us as government evaluations are often conducted independently to meet policy expectations around neutrality. We supplemented the consultations with independent data collection, keeping the work efficient without sacrificing rigour.

Because of their involvement in the review, the program was able to make quick use of the findings, redesigning elements of the Alliance Grants to increase uptake. They called this the Alliance Refresh. In my eighteen years as an evaluator, this was the first time I witnessed such timely and concrete use of evaluation findings to inform program decisions.

Hot Tip:

• Before designing your data collection, ask the program what engagement activities they already have planned and consider joining those alongside your own independent data collection. This saves time, reduces burden on interested parties, and builds trust.

Lessons Learned

If the organization has an evaluation policy or specific expectations for how evaluations are to be conducted, it helps to know those up front and identify potential “gray zones.”   Flexibility is only possible if you understand what is expected and how you can meet those expectations without sacrificing the utility of evaluation.

Rad Resources

• Interested in hearing more? Ed, Jeff and I discuss this review and how the findings were used in Managing for Results in Science, Research and Innovation.
• Jill Chouinard’s article, The Case for Participatory Evaluation in an Era of Accountability, highlights the tension between participatory approaches and the accountability-driven, neutrality-focused culture of public sector evaluation, and makes the case why the former approach still has a place in this context.

Do you have questions, concerns, kudos, or content to extend this AEA365 contribution? Please add them in the comments section for this post on the AEA365 webpage so that we may enrich our community of practice. Would you like to submit an AEA365 Tip? Please do so using the form on our website. AEA365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

There is no shortage of writing on evaluation use, dating back to even before the first edition of Utilization-focused evaluation (U-FE) that I (Michael) wrote back in 1978. But as with anything, much has changed over the past nearly fifty years, and this week is a chance to learn about what is new and top of mind right now. As we dive into this week, however, we wanted to ground our community in how we got here.

The first edition of U-FE was motivated by research that found a lack of intentionality around evaluation use and focused attention on the importance of intended users and uses of evaluation information. A few years later, the standards published by the Joint Committee on Evaluation Standards in 1981 dramatically spotlighted the importance of use. Informing action and supporting evidence-based decision-making is how evaluation contributes to a better world.

Subsequent versions of the book have come out, about every ten years and have added concepts like process use, the personal factor, and principles of UFE. Evaluation Use was one of three major branches on the much debated “evaluation theory tree,” first created by Marv Alkin and Tina Christie in 2004 and updated in 2024. Subsequent attempts to capture the diversity of perspectives in the field have resulted in metro maps (Lemire), gardens (Montrose-Moorhead) and a periodic table (Vaca). Regardless of the metaphor used, evaluation use continues to show up as an important and defining factor in evaluation.

Rad Resource:

In 2000, Valerie Caracelli and Hallie Preskill edited an edition of New Directions for Evaluation on the Expanding Scope of Evaluation Use. In that edition, Karen Kirkhart offered the first integrated theory of evaluation influence, which looked at source, intention and time as three key factors of evaluation influence. A few years later, Mel Mark and Gary Henry built upon what Kirkhart had offered, with a framework “to capture key mechanisms through which evaluation may have its effects.”

Over the past twenty-five years, evaluation use and influence have continued to be important in both the theory and practice of evaluation. Utility standards remain an important part of judging the quality of an evaluation. But as the world becomes more dynamic and complex, we are seeing an increase in the need to consider use through the lens of ongoing innovation and adaptation. Gone are the days of waiting five years for a summative evaluation and expecting that to be useful to make decisions or influence policy, when little is the same as it was five years ago. Developmental evaluation has taken off as a way to ensure ongoing use and influence for real-time decision-making and adaptive resilience.

Hot Tip:

This fall, we will release the 2^nd edition of Developmental Evaluation, which offers the ADAPT framework as a way to promote ongoing use and influence of evaluation that is Agile, Dynamic, Actionable, Principled, and Timely.

Over the course of this week, at this fall’s American Evaluation Association conference, and beyond, we look forward to continuing the conversation about how to ensure that evaluation remains useful and influential in these challenging times.

Rad Resource

For more on the history of evaluation use and influence to ground you in this week’s topic, we invite you to listen to conversations with both Michael and Karen available here.

Do you have questions, concerns, kudos, or content to extend this AEA365 contribution? Please add them in the comments section for this post on the AEA365 webpage so that we may enrich our community of practice. Would you like to submit an AEA365 Tip? Please do so using the form on our website. AEA365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Hello, AEA365 readers! We are Alexis Marbach, MPH, PMP, a Senior Research Scientist, and Megan Cotter, MPH, a Research Scientist at NORC at the University of Chicago. We are evaluators and technical assistance (TA) practitioners who have spent much of our careers helping organizations put evidence-based programs and practices into action. Over the years, we’ve come to see evaluation and TA not as separate activities, but as deeply interconnected practices: good TA requires us to continually collect evidence, learn from implementation, and adapt.

Lesson Learned: Skilled TA providers think like implementation scientists.

Technical assistance is sometimes described as helping organizations learn what to do: providing tools, training, coaching, or a roadmap for implementing a program or practice. But the harder and more interesting question is usually, “What will it take to make this work here?”

That question is at the heart of implementation science.

Every organization operates within a different context. Staffing, leadership support, organizational readiness, community needs, resources, workflows, and competing priorities can all affect whether an intervention succeeds. Skilled TA providers therefore don’t simply follow a playbook. They investigate the conditions surrounding implementation and help organizations determine what needs to change.

Hot Tip: Treat TA encounters as opportunities for structured learning.

TA providers have many of the same tools in their toolbox as evaluators: needs assessments, surveys, interviews, focus groups, observations, feedback forms, implementation data, and facilitated reflection.

The important shift is to use these tools not simply to document what happened, but to ask implementation questions:

• What is helping implementation move forward?
• Where is implementation getting stuck?
• Are the barriers related to the intervention, the organization, the people involved, the broader environment, or something else?
• What assumptions did we make that aren’t holding true? What should we change as a result?

A needs assessment becomes more than a startup activity. A feedback form becomes more than a satisfaction measure. Together, these data create a continuous learning loop that can guide TA throughout implementation.

Lesson Learned: Adaptation isn’t necessarily a departure from evidence.

One of the most important roles TA providers play is helping organizations decide what to adapt and what to preserve.

Implementation rarely unfolds exactly as planned. A strategy that worked in one setting may be impractical in another. Organizations may need to change workflows, communication strategies, staffing approaches, training, or delivery methods.

The goal isn’t rigid fidelity to a playbook, nor is it adaptation without boundaries. The goal is to understand the intervention’s core functions—the essential elements responsible for producing the intended outcomes—and distinguish them from what can be changed for a particular setting.

In this way, we can see midcourse correction (or adaptation) as part of implementation rather than evidence that implementation has failed.

Hot Tip: Build evaluation into the TA cycle.

Instead of thinking of evaluation as something that happens after TA has been delivered, we find it more useful to think in a continuous cycle:

Assess -> Implement -> Learn -> Adapt -> Reassess

This creates a different relationship between evaluation and TA. Evaluation doesn’t simply tell us whether the intervention worked. It helps us understand why implementation is unfolding the way it is and what to do next.

And that may be one of the most valuable contributions evaluators can make: helping teams turn evidence into action while the work is still underway.

Alexis Marbach, MPH, PMP, is a Senior Research Scientist at NORC at the University of Chicago with nearly 20 years of experience helping organizations translate evidence into practice through implementation science, technical assistance, strategic planning, and evaluation. Alexis specializes in designing learning collaboratives, facilitating strategic planning processes, and developing implementation-ready tools that help organizations adopt, adapt, and sustain evidence-informed practices.

Megan Cotter, MPH, is a Research Scientist at NORC at the University of Chicago with more than 10 years of experience conducting and managing primary research, training development, and evaluation tasks, including qualitative and evaluation design, data analysis and synthesis, and dissemination of findings. She supports research across all facets, including project management, data collection and analysis, instrument development, deliverable creation, and client interaction.

Do you have questions, concerns, kudos, or content to extend this AEA365 contribution? Please add them in the comments section for this post so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please do so using the form on our website. AEA365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.

Hello, I am Titilayo Ogunbambi, a community psychologist and evaluator affiliated with National Louis University, Chicago, Illinois. I spent the past decade supporting survivors of intimate partner violence (IPV), engaging in prevention efforts, and evaluating social, cultural, and relational drivers influencing IPV across different contexts. More recently, I had the opportunity to examine the challenges and barriers impeding IPV help-seeking and the use of available formal services among African immigrant women in the United States by interviewing frontline service providers, advocates, and coalition members at the United African Organization in Chicago. The findings inform a culturally responsive framework for co-creating IPV services (See Figure 1 below) and provide evaluators with practical ways to understand reasons why existing systems fall short of addressing IPV and what is needed to design programs that genuinely meet the needs of vulnerable communities.

Lesson Learned

1. IPV is commonly characterized by patterns of control, fear, coercion, silence, physical, emotional, or psychological harm. These factors may appear similar across cultures; however, the forces that sustain them are often culturally specific. In many African family contexts, control is reinforced by an abusive partner in addition to broader systems of authorities like family elders, family lineage, expectations of male leadership, family honor, faith, or traditional norms that normalize male dominance. Violence and abuse in these settings are inseparable from displacement, identity, and the fracturing of gender roles that migration brings, requiring responses that account for both IPV and post-migration-related realities.

For evaluators: IPV is not only an individual relationship problem; IPV is also a community problem with cultural risk and protective factors. Tools that move beyond individual-level definitions of abuse to also include cultural structures are needed to remove some of the current barriers to help-seeking.

2. Psychosocial and structural barriers are often not simply secondary effects of IPV; they affect how women interpret danger and whether safety feels attainable at all. This framework shows that IPV barriers are neither purely external nor purely internal. Rather, barriers are embedded in broader social worlds and systems that determine what women believe is possible when experiencing abuse. Leaving, reporting, or disclosing IPV carries high social and cultural costs, making help-seeking feel risky even when services exist.

For evaluators: Safety means different things depending on who defines it, for whom, and within what cultural context. Therefore, a culturally responsive evaluation should account for safety as relational, culturally mediated, and socially interpreted.

3. IPV service delivery requires a new lens that addresses the structural gaps created when formal institutions overlook survivors’ emotional and relational needs. The goal is to build a service ecosystem where marginalized and vulnerable women are not forced to abandon their cultural worlds to access safety but are supported within them. This community-informed pathway model calls for infusing existing systems with cultural relevance and responsive practices co-created with the communities they serve. Because these dynamics extend beyond African immigrant communities, the model may also offer a useful foundation for other immigrant populations navigating similar barriers.

For evaluators: Low program utilization is often not a client problem. It is more of a systems design problem that signals an opportunity to redesign systems around the strengths, priorities, and cultural realities of the communities they are intended to serve.

The Culturally Responsive Framework for Co-Creating Intimate Partner Violence (IPV) Services, developed directly from this evaluation, maps the interconnected forces shaping African immigrant women’s experiences of IPV and points to what culturally responsive service delivery looks like in practice. The arrows represent barriers that evaluations need to account for, and the pathways at the bottom reflect provider-informed strategies that offer a grounded starting point for co-creating services with the communities they serve.

Figure 1 

My full dissertation includes a six-category thematic analysis and illustrative service pathway and is available here:  Digital Commons at National Louis University.

The question I carry into every evaluation room: Are we designing interventions for the people we imagine or for the people who actually exist, rooted in community, shaped by culture, and making decisions our frameworks have not yet thought to ask about?

Do you have questions, concerns, kudos, or content to extend this AEA365 contribution? Please add them in the comments section for this post so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please do so using the form on our website. AEA365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.

Hi! I’m Sheila B. Robinson, Ed.D., owner of Custom Professional Learning, LLC. While I’m a program evaluator, my main lane is helping people design rave worthy workshops, courses, trainings, and meetings. I’m passionate about high-quality teaching and learning and bring that lens to all my work.

As evaluators, we spend loads of time and energy helping others learn from their programs, their data, and their outcomes. We owe our own sessions that same attention, whether they’re capacity-building workshops, client trainings, quarterly updates, program retrospectives, or funder briefings. Any time you gather people to share information, reflect, or decide together, it’s about learning, and a little thinking drawn from instructional design goes a long way. Here are three tips worth considering before your next gathering.

Hot Tips

1. Start with the outcome, not the agenda

Before you build an agenda, think about the session’s purpose. What should be different because this session happened? What should participants be able to do as a result? Will they build new knowledge or skills? Will the info help inform a decision they need to make? Once the outcomes are clear, plan the topics you need to cover, along with interactive strategies for sharing information so that people will learn it and use it.

2. Right-size what you share

More is not better when it comes to learning. Our working memory can only hold onto a few bits of information at once. Past that, people won’t absorb more. Instead, they’ll feel frustrated and overwhelmed, and will check out. It’s an easy trap for evaluators sharing findings since every data point and bit of context can feel essential, and it’s so tempting to include them all. Before your next session, sort what you have into “need-to-knows” and “nice-to-knows.” Then, let go of those “nice-to-knows,” relegating them to a handout or another session. Trimming your content back makes room for what matters most to land and stick with people.

3. Design for learning, not engagement

Engagement ends when the session ends. Learning shouldn’t. People are pushing hard on the idea of making meetings (and presentations, trainings, etc.) engaging, but what good is engagement if people walk out of the room having learned nothing new? For real learning to happen, whether you’re calling the session a meeting, training, or workshop, you need to incorporate activities that encourage retention and use. It doesn’t have to be complicated! Try building in a few well-crafted question prompts for people to reflect on and discuss in pairs or small groups. Help them make connections from the new material you’re sharing to what they already know. Give them opportunities to talk, write, and share as they’re learning. Use sticky notes (physical or virtual) to generate ideas individually and together, then order or categorize those ideas and make sense of them. For learning to endure, people need to process new information, connect it to their prior knowledge, and practice applying it. So many simple techniques can give people opportunities to do this.

Rad Resources

I frequently write LinkedIn articles related to this topic:

• Three Key Decisions That Shape a Successful Workshop
• Too Much Content, Not Enough Learning: What to Leave Out of Your Workshop or Course
• Beyond Information: How to Design Workshops That Create Action 

My favorite books that support designing any kind of session with people:

• The Art of Gathering: How We Meet and Why It Matters by Priya Parker
• The Power of Moments: Why Certain Experiences Have Extraordinary Impact by Chip and Dan Heath
• Brain Rules: 12 Principles for Surviving and Thriving at Work, Home, and School by John Medina

Do you have questions, concerns, kudos, or content to extend this AEA365 contribution? Please add them in the comments section for this post so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please do so using the form on our website. AEA365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.

I’m Gene Shackman, Applied Sociologist and author of the Beginners Guide to Evaluation, and I’m Elizabeth DiLuzio, Lead Curator of AEA365.. Welcome to part two about non-probability sampling. In the first blog, I talked about quota sampling, used when administering a survey. The other methods are used after the survey has already been administered.

Weighting after data collection

A second way to make a non-probability sample look like the population is post-survey weighting. As Freese and Jin (2025) put it, participants are assigned different weights after data collection to match known population totals on different characteristics. If you have population data on a variable, you can weight your sample to match the population on that variable.

The American Cancer Society used this approach in a recent multi-state tobacco study. The team surveyed 12,300 adults across 17 states in two waves in 2023 and 2024, drawing the sample from a national opt-in opinion panel whose members signed up through email or online marketing. The researchers weighted each state sample to match Census demographics on age, gender, race and ethnicity, education, marital status, and household size, along with smoker status. They reported that tobacco use trends in the weighted sample tracked national probability surveys, and that demographic distributions matched expected population estimates.

That study used straightforward demographic weighting, but the broader literature describes a wider menu. Arletti, Tanturri, and Paccagnella (2025) review the main approaches.

Raking adjusts weights iteratively until the sample matches the population on each weighting variable’s marginal total. It is simple and works when you only have marginal population totals, but it does not capture interactions between weighting variables.

Propensity score adjustment pairs a non-probability sample with a probability sample, estimates the probability that each observation belongs to the non-probability sample, and uses that probability to reweight. Pollard, Robbins, and Griswold (2026) demonstrate the approach with a Twitter sample and an AmeriSpeak probability sample. The unweighted Twitter sample differed from the probability sample on 24 of 27 political attitudes. After propensity score weighting on demographics, technology use, and political ideology, only 2 of 27 differences remained statistically significant. Political ideology did most of the work; excluding it left 14 attitudes still significantly different. Methodologists are also pushing the framework further. Liang and Wu (2026) propose a survey-weighted propensity score weighting framework for causal inference that combines inverse propensity weights with survey weights to address confounding and selection bias at the same time.

Modeling approaches use the non-probability sample to build a model, then apply the model to pre-dict outcomes for the rest of the population. Multilevel regression and post-stratification (MRP) is the most common version. MRP fits a hierarchical model on the sample, then weights predictions by the size of each demographic cell in the population. The hierarchical structure lets the model borrow information across similar cells, which helps when some cells contain few respondents.

Other approaches include statistical matching, which pairs each probability sample observation with its closest non-probability match, and doubly-robust estimation, which combines propensity scoring with modeling so the estimate stays valid as long as one of the two components is correctly specified.

Across all these methods, the choice of variables matters more than the choice of method. Both Freese and Jin (2025) and Arletti and colleagues reach the same conclusion: picking the right weighting variables drives accuracy more than the technique. None of these methods can correct for selection on a variable you did not measure.

Where the field is heading

Research on non-probability sampling is moving fast. Recent reviews of bias-reduction methods are available here and here. Several journals focus on survey methodology, and a useful list is here.

Non-probability sampling is here to stay. The cost and speed advantages over probability samples are too large to ignore, and the methods for adjusting them are improving. But probability samples are not going away either. As Wu argued in a January 2026 Survey Statistician debate, probability samples or census data remain the most crucial ingredient of any defensible statistical analysis of non-probability samples. You need a benchmark to know whether your adjustments worked.

Hot Tip

Gene curates a website of free resources for social research methods. My page on survey sampling has a section on non-probability sampling.

Do you have questions, concerns, kudos, or content to extend this AEA365 contribution? Please add them in the comments section for this post so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to AEA365@eval.org. AEA365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.

Hi! We are Andrew Grillo-Hill, Joshua Valcarcel, Kimberly Nguyen, Karen Melchior, Corynn Del Core, and Rasha Elsayed, the N3 evaluation team within WestEd’s STEM Evaluation Unit. The STEM Evaluation Unit serves as external evaluators for dozens of STEM education projects and programs each year, including for multiple NASA Science Mission Directorate Science Activation 2.0 projects, or SciAct for short.

As evaluators, we wear the hats of being critical friends and facilitators. Today, we would like to share with you one tool we use to share leadership and decision-making responsibilities with our program partners: the Evaluation Debrief tool and process.

We developed this process while serving as external evaluators for NASA’s Neurodiversity Network (N3), a five-year program that provides pathways to NASA participation and STEM employment for neurodivergent learners. N3 involves multiple partner organizations, so transparent communication and shared decision-making were essential from the start.

Here is how the Evaluation Debrief process works. First, we collect and analyze our evaluation data and draft the report. Then, we fill out a debrief form that summarizes key findings alongside our recommendations. We share both the report and the form with our program partners, giving them time to process the findings. Next, we schedule a debrief meeting with relevant partners to discuss and document their reactions and collaboratively decide on the next steps directly in the form. Each of these collaborative and formative meetings has ended with at least one call to action which provides a new iteration of the program for the next cohort, and the cycle begins again.

To illustrate, in Year 1 of the project, our evaluation found that mentors needed more student-specific guidance given the wide range of needs among neurodiverse interns. Through the debrief process, program partners decided to implement “intake meetings” with each intern, their mentor, and an N3 staff member in addition to bimonthly check-ins. This concrete change came directly from the structured dialogue facilitated by the debrief, not from evaluators telling partners what to do, but from partners making informed decisions based on evaluation data.

Rad Resource:

We are sharing a template of our Evaluation Debrief Feedback Form that you can adapt for your own projects. The form includes three columns: Evaluation Recommendation/Finding, PI Response/Question, and Action Taken, along with reflection questions about the timing and usefulness of the findings to give us –the evaluators –feedback too.The form, by design, is simple and concise to allow for  less more open-ended and less “dictated”  conversations during the debrief discussions.

A PDF of the first page of the form, with two paragraphs of instructions for program partners.

A PDF of the second and last page of the form with a blank table to fill out and two open-ended questions for feedback on the evaluation.

Lessons Learned:

As the N3 program nears our end date this year, we have been able to look back at how this process shaped the partnership that allowed for a program that was so rewarding to be a part of. Three things, especially, stand out.

1. The form created a documented trail of data-informed decision-making that partners could revisit over time.
2. Transparent documentation of the decision-making built trust across organizations.
3. The process gave program partners collective ownership over program modifications.

Overall, the Evaluation Debrief process contributed to more efficient, equitable, and transparent shared leadership throughout the life of the project.

Do you have questions, concerns, kudos, or content to extend this AEA365 contribution? Please add them in the comments section for this post so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to AEA365@eval.org. AEA365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.

Hello! I’m Arthur Hernández, an evaluator and educator whose work focuses on evaluation, community engagement, and the forms of knowledge we authorize as credible. I want to address a persistent and perhaps unvoiced misunderstanding: the idea that “lived experience” is either redundant or somehow a threat to scholarly expertise.

Lesson Learned: Lived experience is not merely “experience”.

The phrase “lived experience” can sound redundant because, in one sense, all experience is lived. All learning requires experience of some kind. We read, listen, observe, analyze, compare, infer, and test meaning through experience. Even academic expertise is built through experience: disciplined study, repeated inquiry, methodological practice, critique, and participation in scholarly communities.

But that does not make all experience the same. Lived experience, as I use the term, refers to first-hand encounter with a phenomenon, condition, institution, place, policy, culture, or pattern of treatment. It involves more than exposure to information about something. It includes the situated, embodied, relational, and often consequential experience of being in contact with the thing itself. It often requires active engagement, interaction, navigation, vulnerability, interpretation from within, and direct participation in the meanings and effects being evaluated.

A person may study poverty, disability, schooling, immigration, policing, public health, or community violence with rigor and care. That expertise is real. But it is not the same as having had to live, negotiate, resist, survive, make meaning within, or be shaped by those realities.

Hot Tip: Stop treating lived experience as anti-intellectual.

One of the most unhelpful moves in evaluation is to position lived experience against academic or scholarly expertise, as though recognizing one diminishes the other. That is a false opposition. Lived experience does not automatically make someone correct, any more than a degree automatically makes someone wise or expert. But lived experience can be a worthy candidate for expertise because it may disclose meaning, consequence, feasibility, harm, benefit, and legitimacy in ways that external analysis alone may not.

The issue is not whether lived experience should replace technical expertise. It should not. The issue is whether evaluation can responsibly claim to judge merit, worth, significance, or consequence while dismissing the knowledge of those most directly implicated. It cannot.

Lesson Learned: Authenticity is not the same as accuracy.

Accuracy asks whether a statement corresponds to evidence. Authenticity asks whether an interpretation is faithful to the meaning, situated reality, and lived consequence of what is being described. A finding can be technically accurate and still be incomplete, distorted, extractive, or illegitimate to those whose lives are being represented.

This is why lived experience matters across epistemological and ontological traditions. Positivist evaluators can value it for construct clarity, measurement relevance, and interpretation. Pragmatists can value it for use, consequence, and fit. Critical, Indigenous, feminist, and participatory evaluators can value it because it challenges whose knowledge counts and who has authority to interpret reality. Across these traditions, lived experience strengthens evaluation because evaluation is never conducted from nowhere.

Across these traditions, lived experience strengthens evaluation because evaluation is never conducted from nowhere.

Rad Resource: Ask who is appropriately positioned to participate.

A better question than “Who has data?” is “Who is positioned in relation to the purpose for/of evaluation in ways that make their participation appropriate, necessary, and legitimate?” Community positionality matters. Engagement is not a courtesy added after technical work is complete. It is part of determining whether participation is meaningful, whether interpretation is warranted, and whether evaluative judgment can be considered legitimate.

Lived experience is not the enemy of expertise. It is one of the ways evaluation becomes more authentic, more accountable, and more worthy of the communities it claims to serve.

Do you have questions, concerns, kudos, or content to extend this AEA365 contribution? Please add them in the comments section for this post so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to AEA365@eval.org. AEA365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.

Hello, AEA365 readers! We are volunteer AEA members of the Conference Advisory Working Group (CAWG), with experience across TIGs, task forces, working groups, and other parts of the association. We assist with the structure, operations, and evaluation of the annual conference. 

In January 2026, AEA’s Board established the Financial Vitality Task Force and charged the Executive Director and CAWG with reducing conference-related costs to support AEA’s long-term sustainability. We were also asked to streamline Eval26 to strengthen program coherence, improve the attendee experience, and expand cross-topic learning. 

Redesigning a major conference while delivering it has been challenging for everyone involved. CAWG has been learning as we go, and AEA members have been navigating major changes to familiar parts of the conference, including streams and changes to proposal submission, review, and selection. Nearly every part of the redesign has raised questions, concerns, resistance, and helpful feedback. 

Lessons Learned

Make the Work Visible and Remain Open to Feedback

People are better able to engage with change when they can see the work behind it. We have tried to explain not only what was changing, but why, through conversations with TIG leaders, emails to AEA members, and information on the conference website. 

We have repeatedly heard that this transparency and openness have been strengths. For us, openness means listening carefully, being honest about what can and cannot change now, remaining curious about what we may have missed, and using feedback to inform longer-term changes.

Design for Effective and Efficient Collaboration

Previously, reviewers were commonly recruited through TIGs, although only about half of AEA members belong to at least one TIG. For Eval26, we opened the reviewer application to the full membership, and more than 250 people volunteered. Nearly all TIGs were represented, including nearly 50 individuals who were not yet members of any TIG. 

We also became more intentional about which work required a meeting. CAWG members drafted and reviewed materials asynchronously, giving people time to meaningfully contribute. Meetings were most useful for working through disagreement, connecting perspectives, and collective sensemaking. 

Stream-team members followed a similar rhythm. New for Eval26, stream teams looked beyond individual proposal scores to help shape a strong, coherent program within each stream by considering quality, duplication, balance, and connections across topics. Members reviewed independently, then came together to deliberate and make recommendations. Collaboration improves when asynchronous and synchronous work each have a clear purpose.

Proof the Process, Not Just the Products

Just as we proofread a document, we also need to “proof” the processes through which people submit information, review proposals, deliberate, and receive communications. One tool we found useful is a pre-mortem: imagine that a process has failed and work backward to identify what could have caused it. Asking “What is most likely to go wrong?” gives people permission to challenge assumptions and surface concerns before implementation. Then, test the process from beginning to end using realistic examples viewed from each participant’s perspective. Testing will not guarantee perfection, but it can reveal gaps that are otherwise easy to miss.

Hot Tip: Create a Space to Continue the Conversation

Join us for the Eval26 Reflections and Looking Ahead to Eval27 Town Hall on August 13 from 2:00–3:00 p.m. ET (register for the town hall).

We will share what we are learning through feedback and evaluation efforts, what those lessons may mean for Eval27, and upcoming design thinking and listening activities. Bring your questions, reactions, and ideas for what comes next!

Do you have questions, concerns, kudos, or content to extend this AEA365 contribution? Please add them in the comments section for this post so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to AEA365@eval.org. AEA365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.