Toiling somewhere in the research ecosystem right now, there is a researcher who will save lives. Likely, this scientist is publishing in mid-tier journals and has never had a paper published in a highly prestigious venue. Yet, their work is already being cited by the scientists who will build on it and translate their work into lifesaving interventions. We know this because we can see them in the data, if we care to look. Under the evaluation systems most institutions use today, this researcher is largely invisible. Yet this outcome may occur despite our evaluation systems rather than because of them.

Scientists aspire to evaluation systems that are meritocratic and fair. This raises the question, fair to whom? Science has many stakeholders from enterprising researchers to taxpayers who would like to see science stimulate the economy through innovation, and patients waiting for new or better treatments for disease. At many institutions we have evaluation systems that primarily acknowledge prestige bestowed by the branding of journals and conferences. This system arose from the creation of Cell in the 1970s, marketed as the prestige journal where authors would have the last word on a subject, and later extended to families of prestige journals operationalized by the Journal Impact Factor.

By definition, this system works well for those who have survived the academic gauntlet and now shape self-reinforcing evaluation systems. However, we also know that this process overlooks tens of thousands of researchers who have never published in prestige journals but have authored equally influential work. It fails to acknowledge important contributions to clinical translation that save lives. Downstream economic innovation and patenting are nearly invisible in academic reward systems. This is problematic because it is ultimately unfair, not only to overlooked researchers, but to taxpayers who want innovation to drive a growing economy, and to patients waiting for a new hope for currently intractable diseases.

Hot Tip

Ultimately, the evaluation measures we choose shape the science we get. To ensure fairness to all the stakeholders of the research enterprise, these should intentionally reward broadly valued outcomes, rather than producing these as a side-effect of prestige. Fortunately, scientific knowledge networks can identify who is publishing highly influential work outside prestige venues, which discoveries led to inventions, what basic research translated into clinical knowledge, and how novel therapeutics were developed. In addition, these impact-anchored evaluation measures can also be used in forecasting, to flag important bodies of work that look like they will stimulate the kinds of applied research outcomes that resonate with diverse stakeholders.

When discussing fairness in evaluation, the ugly side of research integrity problems cannot be overlooked. These are most visible in retracted papers, but likely stem from structural factors. Misinformation propagated from unreliable published work harms all the stakeholders in the research ecosystem. Here too, scientific knowledge networks may have a role to play. Our lab is studying whether the same methods that can forecast positive outcomes like breakthroughs or future FDA-approved therapeutics, can also help identify areas of heightened research integrity risk. Such methods might be able to identify structural components of research integrity risk that underpin misconduct.

The anticipated deluge of AI-generated articles can frame how these contributions are viewed. Are they a massive research integrity problem? Will they increment the frontier of knowledge somewhat, but ultimately remain self-referential? Or will these lead to faster innovation and patenting, and more new disease therapies? Impact-anchored evaluation measures can help us discern.

The American Evaluation Association is celebrating RTD TIG Week with our colleagues in the Research Technology and Development TIG. All of the blog contributions this week come from our RTD TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

For large or complex portfolios, evaluators face a challenge: how can we make sense of large bodies of scientific work without relying exclusively on predefined categories or manual coding?  Our team recently explored this question, creating a topic visualization of National Institutes of Health (NIH)-funded grants using embedding-based language models. Our goal was to understand how scientific topics are distributed across NIH Institutes and Centers (ICs), while experimenting with methods that can capture the relationships within research portfolios at scale.

Many existing topic or content analyses rely on administrative classifications, manually assigned categories, or counts of frequently used terms. While useful, these methods can struggle with interdisciplinary science, evolving terminology, or emerging research areas. Embedding-based approaches offer a more flexible way to represent scientific activity.

Rad Resource

Our approach used PubMedBERT, a language model trained on the biomedical literature (PubMed). Unlike keyword-based methods, embedding models like PubMedBERT look at a text’s language in context, allowing comparison of documents based on meaning. Grants can be grouped because they describe related concepts, even if they do not use the same terminology.

We created embeddings from grant titles and abstracts, clustered grants into topics, then used these results to develop a beta interactive tool, TopicVista, to explore the distribution of topics across NIH ICs. TopicVista revealed both expected patterns and areas of overlap across the NIH funding landscape. Many topics are concentrated within the expected ICs, such as “Cancer Biology” within the National Cancer Institute (CA).  Other topics appeared across multiple ICs (e.g. “Data Management and Bioinformatics”). These areas of overlap may point to opportunities for collaboration across ICs, or overlap may raise useful questions about how ICs delineate their distinct roles and priorities. Our tool, code, and data are freely available here.

For evaluators, embedding-based topic models create new possibilities. First, they can reveal how research areas relate by content rather than administrative category, helping evaluators identify overlap, fragmentation, convergence, and gaps across organizations or initiatives. Second, these approaches can support analysis of interdisciplinarity and emerging science by surfacing connections conventional classifications may miss. Third, visual topic maps can make complex funding landscapes easier to communicate, helping decision makers and stakeholders more intuitively understand research activity.

Importantly, these methods do not replace substantive expertise or evaluative judgment. Models such as PubMedBERT reflect patterns in scientific language; interpretation still requires domain knowledge, validation, and methodological transparency. But these approaches expand the evaluator’s toolkit, especially for working with larger and less structured datasets.

For the RTD TIG community and other evaluators, the broader opportunity may lie in how advances in natural language processing can strengthen evaluation practice itself. As computational methods become more accessible, embedding-based analyses offer evaluators new ways to examine research systems, organizational strategy, and scientific ecosystems at a scale that was previously difficult to achieve.

TopicVista represents an initial effort to explore NIH research portfolios using embedding-based topic models. The topics and visualizations have undergone limited manual review, and we view this as the beginning of an iterative process rather than a finished product. We welcome feedback on the tool, its interpretations, and potential applications, and would be delighted to connect with others interested in refining these approaches or collaborating on future work.

The American Evaluation Association is celebrating RTD TIG Week with our colleagues in the Research Technology and Development TIG. All of the blog contributions this week come from our RTD TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Since our program was formalized in 2014, we have worked to better understand the impact of our grantmaking. What began as an informal process of cataloguing grantee outputs evolved into an internal database that tracks publications, conferences, students supported, and additional funding raised through Sloan-supported research. The database serves as a light-touch tool for monitoring measurable outcomes while helping us identify broader trends over time, and has also informed our strategic planning process.

We are pleased to share the lessons we learned along the way about incorporating assessment into routine operations, using the information strategically, and adapting the process over time.

Time is Precious, for Both Grantees and Funders

We initially needed something we could build and maintain with limited staff time and resources available. A simple Excel spreadsheet has worked well for us because it is organized, straightforward, and adaptable. We focused on tracking categories most relevant to our program strategy of generating new knowledge, supporting early-career scholars, and connecting research to practice. Starting simple was valuable, allowing us to iterate based on our needs and priorities. For example, we initially wanted to track conference attendees, but it quickly became apparent that would become intractable, so we removed that category.

The process was still time-consuming; establishing the tracker tool took about four months of dedicated time from a single team member, but starting with the simplest option got us insight sooner, prevented us from getting locked into a more onerous process, and gives us the flexibility to move to a more advanced system in the future if our needs change. Ultimately, it is important to be honest about the capacity you have available, but getting going is the most important step.

Hot Tip: Standardize Information Flows but Allow Narrative Freedom

When beginning the tracking effort, our grantee reporting guidelines asked for a loose narrative structure that included general descriptions of outputs and progress updates relative to grant metrics and goals. The level of detail varied greatly between reports, sometimes requiring us to fill in the gaps where information was not provided.

To facilitate the tracking process without adding undue burden on grantees, we limited the amount of new information we asked for and then clarified and updated our guidelines to better align with how we extract information. We now ask for information on written outputs and students in table format with categories that match those used in the database while also retaining the overall narrative structure of the reports to allow grantees freedom in their responses and ensure we don’t miss key qualitative updates.

Nothing Replaces Reading the Reports

In order to understand the full impact of a funded project, quantitative outputs require qualitative context to fully situate outcomes within the broader research landscape. That remains true. We learn so much from reading those report narratives, from cutting-edge research results to contemporary policy impacts. To fully connect outputs to impact, it is critical to have context that can only be provided by making sense of full reports.

Final Thoughts

Retaining flexibility in the reporting process is important so funders can be responsive to not only their needs but also those of their grantees. Perhaps the most important lesson we learned is that output tracking is just one component of a well-stocked portfolio of impact assessment techniques and approaches.

The American Evaluation Association is celebrating RTD TIG Week with our colleagues in the Research Technology and Development TIG. All of the blog contributions this week come from our RTD TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

I’m Andrew Feig – Senior Program Director at Research Corporation for Science Advancement (RCSA). At RCSA, we have two major programs, Cottrell Scholars and Scialog (as well as a nascent RCSA Fellows Program), but for this discussion, I will focus on Scialog. Scialog is a contraction of Science + Dialog. The program brings together groups of scientists (Scialog Fellows) who do not yet know each other but should. After a structured, data-driven process to introduce Fellows to one another and 1.5 days of facilitated discussions, participants spend the last half-day forming teams and writing proposals. Themes typically run for three years, and Fellows return year-on-year for the duration of the initiative, with additional rounds of grant-making each year. RCSA’s goal is to fertilize new areas of science, seed new interdisciplinary collaborations, and help early-career faculty become leaders in the thematic area, where their bold ideas change the direction of scientific thinking. 

At one level, our Board wants big outcome assessments. What were the ground-breaking scientific advances that derived from these grants? How did these grants change the field? Unfortunately, since we fund early-stage ideas, it often takes 5, 10, or even 15 years for the most successful projects to evolve and yield that level of impact. These stories do not provide actionable insights to guide the program and thus have only modest value in my opinion. Collect them, but don’t spend too much time or resources here.

Hot Tip

Far more important are the early hallmarks of success that we can use as proxies for what may eventually come from these teams. Do the collaborations persist? What occurred during Scialog that helped the teams gel? Was there sufficient early success that the team sought or acquired additional funding based on data our funding helped them obtain? We collect early hallmark data from structured grant reports, and supplement those reports with annual analyses of every past Scialog – back to its inception more than 15 years ago. Using APIs, we pull publication histories of every participant (more than 2000 to date). We tag papers co-authored by 2 or more Fellows who met at Scialog, noting whether or not they received funding or wrote a proposal together. We use these papers to track the long-term network effects of the community we seeded. It turns out that many of these papers come from teams that never received a grant and thus would never have submitted a progress report. They liked the idea they proposed and chose to pursue it with their colleagues anyway.

Even more important, however, are the questions about the activities during Scialog. What activities effectively build trust between Fellows? What motivates two individuals to choose to collaborate just 24 hours after having met? Through a project with Prof. Danny Abrams (Applied Mathematics @ Northwestern), we have measured the interpersonal interactions during our meetings, the nature and role of facilitation in discussion groups, and the dynamics of group conversations that break homophily and drive creative thinking. These studies have led to changes in our conference schedule, modifications of our facilitator training, and optimization of the event in transformative ways. This is the type of assessment I lean towards: INTENTIONAL – based on data collections we build into the planning and execution of our meeting; it is part of what we do not an add-on;  ACTIONABLE – allowing us to use it to refine our convening formats; and IMPACTFUL – providing a feedback loop that lets us tell the story of how we use assessment in the day-to-day work of our foundation and pursue continuous program improvement.

The American Evaluation Association is celebrating RTD TIG Week with our colleagues in the Research Technology and Development TIG. All of the blog contributions this week come from our RTD TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

NIH’s central responsibility is to support the scientific community, primarily through competitive funding for research. That work is done in partnership with experts pulled from the community itself: previously funded researchers who review grant applications and advise NIH institutes and centers. Complementary and integrative health (CIH) investigators, a subset of scientific researchers holding specialized certifications (osteopathy, acupuncture, naturopathy, and others), do not receive NIH funding or participate in these activities at the same rate as non-CIH investigators. The CIH community was advocating for increased participation, and leadership at the National Center for Complementary and Integrative Health (NCCIH) needed to understand why the deficit existed before deciding how to address it.

Developing a composite metric of success

CIH research has long carried a perception in some quarters that it isn’t as “successful” as mainstream research, which some would argue explains the deficit. To rule out the research-quality explanation, I needed to compare how CIH and non-CIH investigators performed once they were in the NIH system.

Success for a scientific investigator hinges on NIH funding, publications, and impact on subsequent studies or clinical trials. A true success metric combines all of these and accounts for career stage. That metric didn’t exist in the data, so I built one. I used my subject matter expertise to combine the underlying variables (grant review metrics, publications, scientific impact, time) and sort investigators into five success levels (Unsuccessful, Too Early, Mildly Successful, Moderately Successful, Highly Successful). This was time-consuming and depended heavily on my expertise, which limited its transferability. So, I used the labeled dataset to train a Random Forest classifier, a machine learning model that learns your definition once and applies it consistently to new data.

Changing perceptions to identify the underlying issue

I assembled ten years of NCCIH application, funding, publication, and impact data. I sorted CIH and non-CIH investigators into the success buckets and compared the distributions. They were almost identical. Once funded, CIH investigators were performing at the same level as their peers. The gap wasn’t about the quality of the research.

So, I looked earlier in the pipeline and found that fewer CIH investigators were submitting applications to NCCIH in the first place, by a wide margin. This mattered because grant applications are how an investigator gets into the NIH system. If you don’t apply, you aren’t in the system, and you can’t be selected for funding, review panels, or advisory committees. That explained the deficit the community was noticing and gave leadership a clearer picture of how to address it. The conversation shifted from dedicated funding mechanisms for already-competitive researchers to supporting an application pipeline: recruitment, first-time applicant support, and removing barriers to submitting.

Lesson Learned: solve the real problem, not the assumed one

Stakeholders bring evaluators their problems, but they also bring perceptions about what’s causing them. Sometimes the perception is right and the evaluation confirms it. Sometimes it isn’t, and the evaluation’s job is to find what’s actually going on. Doing that work well usually means three things: testing the perception against the data instead of accepting it as given, building new metrics when the data doesn’t answer the question directly, and following the data to the real issue rather than the one you started with. When evaluation does all three, the solution gets aimed at the problem that actually exists.

The American Evaluation Association is celebrating RTD TIG Week with our colleagues in the Research Technology and Development TIG. All of the blog contributions this week come from our RTD TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Research, technology, and development evaluation sits at the intersection of evidence and decision-making. Whether we are assessing grant programs, examining research portfolios, understanding scientific impact, or informing strategy, our goal is ultimately the same: helping organizations make better decisions based on evidence rather than assumptions.

That sounds straightforward, but anyone who has worked in evaluation knows the challenge. We rarely have perfect data. Stakeholders often arrive with strong beliefs about what is happening and why. The systems we study are complex, interconnected, and constantly changing. And sometimes the questions we are asked are not the questions that most need answering. As we launch RTD TIG Week, it is worth reflecting on a few practices that consistently strengthen evaluation, regardless of the setting, methodology, or sector.

Lessons Learned

1. Start with the question behind the question

The first problem presented to an evaluator is not always the real problem. Before collecting data or selecting methods, spend time understanding what decision needs to be informed and what assumptions may be shaping the conversation. Some of the most valuable evaluations emerge when evidence challenges prevailing beliefs and redirects attention toward a more important issue.

2. Design for learning, not just reporting

Evaluation is most useful when it supports continuous improvement. Consider how information will be used before deciding what to measure. Data collection that is built into programs and organizational processes often produces more actionable insights than assessments conducted only at the end of a project.

3. Build systems that can grow with your needs

Organizations frequently begin with ambitious plans for data collection and assessment. In practice, sustainable systems are often the ones that start simple, focus on the most important information, and evolve over time. A modest but consistently maintained tracking system can generate far more value than a sophisticated system that becomes too burdensome to sustain.

4. Look for patterns, connections, and context

As research portfolios and programs grow in size and complexity, evaluators increasingly need ways to understand relationships across projects, disciplines, organizations, and outcomes. Quantitative metrics remain important, but understanding how activities connect and where areas overlap can reveal opportunities that individual measures alone might miss.

5. Keep your focus on impact

Metrics are tools, not goals. Publication counts, funding totals, participation rates, and other indicators can provide useful information, but they are ultimately proxies for something larger. Effective evaluation continually asks whether the measures being used align with the outcomes and impacts that matter most to stakeholders and society.

These ideas are not new, but they remain central to effective evaluation practice. Throughout RTD TIG Week, we will explore how evaluators and research organizations are applying these principles in different contexts, using a variety of approaches and methods. While the settings may differ, a common thread runs through all of them: a commitment to using evidence thoughtfully, asking better questions, and helping organizations learn.

We invite you to join us throughout the week as we examine the evolving tools, challenges, and opportunities shaping research evaluation today. We hope the discussions spark new ideas for your own work and encourage you to share your experiences with the RTD TIG community.

The American Evaluation Association is celebrating RTD TIG Week with our colleagues in the Research Technology and Development TIG. All of the blog contributions this week come from our RTD TIG members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Hi, I’m Jacqueline Corum (she/her), a Cooperative Extension evaluator at the University of Kentucky and graduate student in the Educational Research Methods and Policy Studies program.

In a previous role, I facilitated a grassroots community and leadership development program at the Brushy Fork Leadership Institute called “People Ready Communities.” This program supported diverse teams of community members as they led projects aimed at improving their communities’ “readiness” to be places where all residents experience safety, acceptance, inclusion, and belonging. We did this through capacity building; enhancing local leaders’ ability to make data-informed, collaborative decisions that supported the planning and implementation of short-term community development projects based on their unique local needs and context.

This grassroots model informed my work this semester as I conducted interviews and focus groups exploring how LGBTQIA+ folks in Lexington, Kentucky experience and navigate feelings of safety and belonging in their daily lives. I used this qualitative research development opportunity to improve both technical skill and academic knowledge through incorporating theories on intersectionality, microaggressions, belonging, and aging in place to better understand contributing factors. Despite growing availability and pressures to use AI to support research and data analysis, I coded the data without using AI, reflecting throughout the coding process on my positionality as a researcher and member of the community I am studying. I also think I captured thematic elements relating to queer culture that AI may have missed or represented less authentically. I found Dedoose to be a helpful tool as I went through iterations of open coding, describing and organizing themes, and creating a codebook with excerpts.

My next step is to continue exploring the connections between belonging and community capitals. The relationship between belonging, community capitals and economic impact appears to be a gap in literature. I believe the economic justification for fostering LGBTQ+ safety and belonging effectively compliments the qualitative data on the topic.

Reflection

How can we leverage evaluation to foster communities where all members feel safe and that they belong?

Hot Tips

• RESIST outsourcing qualitative data analysis to AI.
  •  In an interestingop ed from Inside Higher Ed, Andrew Gillen points out four main points: the researcher is just as important as the research; AI is not neutral; adoption of AI tools can have a negative impact on the training of new researchers; and AI can’t safeguard data like human researchers.
  • Claire Moran advised that ethical awareness requires oversight. “AI can hallucinate citations, reproduce bias, and generate content that sounds right but is wrong.”  They added Roberts et al. “caution that LLMs imitate language patterns, not truth, with AI’s ‘human like’ qualities resulting from statistical predictions and not comprehension. Without human review, misleading or unethical outputs can slip through.”

Rad Resources

• Dedoose is a low-cost, user-friendly tool for manually coding qualitative data themes, creating theme visuals, and data reporting.
• HRC’s Municipal Equality Index provides a rubric and analysis of city’s LGBTQIA+ related policies
• PFLAG’s Guide to Being an Ally to LGBTQ+ People
• AARP Livable Communities resources support communities to increase their accessibility and safety for ALL, especially for the disability and aging communities.
• Welcoming America is a national nonprofit with resources focused on supporting communities to become more welcoming and inclusive of immigrant populations.

If you are interested in learning more or joining my work, please reach out to me. I would like to identify potential collaborators interested in supporting a joint research proposal, data collection and analysis, and/or publication development.

The American Evaluation Association is hosting LGBTQ+ Voices in Evaluation TIG Week with our colleagues in the LGBTQ+ Voices in Evaluation Topical Interest Group. The contributions all this week to AEA365 come from our LGBTQ+ Voices in Evaluation TIG members. Do you have questions, concerns, kudos, or content to extend this AEA365 contribution? Please add them in the comments section for this post on the AEA365 webpage so that we may enrich our community of practice. Would you like to submit an AEA365 Tip? Please send a note of interest to AEA365@eval.org. AEA365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.

Hi, I’m Laura Pinsoneault and Kristen Gardner-Volle from Evaluation Plus. We served as the evaluator on a PCORI award that brought community leaders, clinicians, and researchers together to understand and reduce cancer disparities among transgender, nonbinary, and gender-diverse (TGD) people. We worked with the Community and Cancer Science Network (CCSN), which starts with a simple belief: how we connect drives change.

Lessons Learned: Collaboration Is Health Equity

When we began exploring why TGD people experience worse cancer outcomes, we quickly realized the issue wasn’t just missing data or bias, it was also how knowledge itself is produced. The CCSN model treats relationship-building as research infrastructure. We brought together TGD community leaders, clinicians, and scientists as equals to map root causes of cancer disparities from systemic discrimination to hormone access.

To make that collaboration real, we started with team design. We used a MoSCoW matrix (“must have, should have, could have, wish for”) to identify whose perspectives were essential—community advocates, researchers, clinicians, and people with lived expertise. This structure balanced expertise, experience, and identity before the project began. For evaluators, this means that partnership design is part of an equity practice: it shapes what knowledge emerges and whose wellbeing is centered.

Hot Tips: Use Creative Tools to Transform How People Think Together

We mixed community members, clinicians, and researchers into small groups to design an Album Cover—titling it to show what the world would be like if it was free from disparities and three “title tracks” that described how to get there.

It was playful and energizing—but also deeper. As people built their albums, they began to see their expertise in new ways. Scientists talked about belonging. Community partners talked about systems change. The exercise didn’t just give everyone a voice; it reshaped how we understood our knowledge and its limits. It moved us from parallel expertise to shared insight and bonded the team in ways no formal meeting could.

Cool Tricks: Move from Vision to Action

Once our shared vision took shape, we used a Consensus Workshop to turn ideas into action. The process blends the Focused Conversation method with five stages of collective synthesis. It balanced dialogue, surfaced wisdom and turned inspiration into action.

Here’s what emerged:

Rad Resources

• Our CCSN Toolkit includes tools, team-building guides, and templates to help others design transdisciplinary partnerships grounded in deep equity.
• As evaluators for the SAGE LGBTQI+ Advancing Elders Project, we learned from the Global Story Archive where LGBTQ+ elders share personal stories about health, identity, and resilience. For evaluators, these questions matter because they expand what counts as evidence and remind us that the pursuit of LGBTQ+ health equity is inseparable from the pursuit of human dignity. Story-based evidence helps us see the systems that harm and the possibilities that heal.

Reflection

Health equity grows in the spaces where science and community meet, listen, and learn together. Relationship-building is equity work. Creative structure simply gives it the shape it needs to thrive.

This work was developed by a transdisciplinary team including Chandler Cortina, MD; Tobi Cawthra, MPH; michael munson; Caleb Weinhardt, Andrew Petroll, MD; and Melinda Stolley, PhD, in partnership with FORGE and the Medical College of Wisconsin Cancer Center.

This project was funded through a Patient-Centered Outcomes Research Institute (PCORI) Eugene Washington PCORI Engagement Award (EA #25591). The views presented in this article are solely the responsibility of the author(s) and do not necessarily represent the views of the Patient-Centered Outcomes Research Institute® (PCORI®), its Board of Governors or Methodology Committee.

The American Evaluation Association is hosting LGBTQ+ Voices in Evaluation TIG Week with our colleagues in the LGBTQ+ Voices in Evaluation Topical Interest Group. The contributions all this week to AEA365 come from our LGBTQ+ Voices in Evaluation TIG members. Do you have questions, concerns, kudos, or content to extend this AEA365 contribution? Please add them in the comments section for this post on the AEA365 webpage so that we may enrich our community of practice. Would you like to submit an AEA365 Tip? Please send a note of interest to AEA365@eval.org. AEA365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.

I am Veronica S. Smith, MS (she/they), a community data and evaluation guide and solopreneur of data2insight.

I partner with LGBTQ-serving organizations doing community-driven data collection, analysis, and storytelling that centers lived experiences and everyday realities. This work is informed by The Coalition of Communities of Color’s research justice strategy guidelines:

• Community members are experts.
• Communities are positioned as researchers not just subjects of research and inquiry.
• Communities can conduct critical and systemic inquiry into their own lived experiences.
• QTBIPOC knowledge and expertise can counter dominant cultural narratives

Hot Tips

A community data party is an activity we use in our work to grow trust across ecosystem actors and capacity to conduct critical and systemic inquiry.  A data party is an in-person event where diverse community members collectively analyze and make sense of data, equitable research method for community engagement.

A data party’s purpose is to:

• encourage people to make sense of data together,
• center community members’ in use of data to make policy, law, and/or provide resources to benefit a community,
• gives the community informed decision making.

The potential benefits of a data party are:

• increased community research knowledge and interpretation of data,
• data sense making happens within a community,
• community-backed recommendations to inform the community, policymakers, lawmakers, and funders.

Lessons Learned

Kory Higgs, and I co-facilitated a community data party for The Collaborative, made up of Queer Power Alliance, POCAAN, Entre Hermanos, and Pacific Northwest Black Pride. Our focus was gathering community members to make sense of survey results about economic stability for South King County LGBTQIA+ residents. About 25 folks joined for food, music, community, and conversation.

The roundtable discussions were the event highlight. Attendees reviewed data placemats featuring information and questions about the three focus areas of transportation, healthcare, voting and civic engagement. Conversations reflected on experiences, challenges, and possible solutions.

One party goer said they enjoyed the conversations and want more time to continue those conversations. Another appreciated that findings were shared in plain language and easy to make sense of compared to some research findings that “make me feel dumb.”

The Collaborative is building on this work to co-create a 2027 legislative agenda to improve economic stability for South King County LGBTQIA+ residents.

Rad Resources

These great resources are from Corey Newhouse and Dana Benjamin, leaders of Public Profit and Back of the Napkin (respectively).

• Dabbling in the Data: A Hands-On Guide to Participatory Data Analysis
• Back of the Napkin Downloadable Resources

This Stanford Social Innovation Review article explains how overreliance on dominant data fails communities and how community data provides a solution.

• Community Data Is Trusted Evidence by Mira Mohsini & Andres Lopez

Culturally Responsive and Equitable Data Parties: A Method for Participatory Analysis and Sense-Making in Virtual Spaces

The American Evaluation Association is hosting LGBTQ+ Voices in Evaluation TIG Week with our colleagues in the LGBTQ+ Voices in Evaluation Topical Interest Group. The contributions all this week to AEA365 come from our LGBTQ+ Voices in Evaluation TIG members. Do you have questions, concerns, kudos, or content to extend this AEA365 contribution? Please add them in the comments section for this post on the AEA365 webpage so that we may enrich our community of practice. Would you like to submit an AEA365 Tip? Please send a note of interest to AEA365@eval.org. AEA365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.

Hello! We’re McKenzie Javorka, Patrick Boise, and Kaitlin Fertaly, evaluators at the University of Montana’s Rural Institute for Inclusive Communities.

Montana has major gaps in pre-exposure prophylaxis (PrEP) uptake among people who could benefit from it, especially men and people assigned male at birth (AMAB) who have sex with men/AMAB individuals. To understand why, we partnered with our state public health department to conduct a statewide PrEP needs assessment.

Lessons Learned

We’ve done plenty of needs assessments in Montana, but this one challenged us in new ways, especially when it came to recruiting participants.

Our original plan was straightforward: run a statewide online survey, then invite a subsample of respondents to do follow-up interviews. Given Montana’s size and rurality, an online survey seemed like the most efficient and affordable option. But soon after launching, our survey was flooded with bots and fake responses. Sorting through what was real versus fake or invalid became a major task.

Even the next step, inviting survey participants to follow-up interviews, was tricky. Some scammers went as far as completing phone interviews with false information to earn incentives. It quickly became clear we needed a new approach.

We pivoted and instead began recruiting through community-based organizations, health clinics, and word-of-mouth networks. We distributed flyers, partnered with trusted LGBTQ+ and harm reduction groups, and tabled at a drag show to connect directly with potential participants. It took longer than expected and skewed more urban than we hoped, but these efforts finally got us the participation we needed.

Hot Tips

• Expect bots, AI, and scammers in online surveys. We built in checks like requiring a valid Montana zip code and logic questions, but they weren’t enough. AI-generated and falsified responses can easily slip through. In the end, we had to apply multiple filters after data collection and still had lingering concerns about data validity.
• Be cautious about advertising incentives. Incentives can help with recruitment but publicizing them online can also attract bots and scammers. If you must mention them, consider mailing physical incentives (which require a real address in the study area) or using verification systems before sending digital rewards.
• Partner with trusted organizations and “study champions.” Our most effective recruitment came through LGBTQ+ and harm reduction organizations that directly shared our study with clients. We also relied on highly connected community members who spread the word through social media and personal networks. However, these tactics may not be the most effective for reaching people in the most remote areas or those who aren’t openly LGBTQ+.
• Offer flexible participation options. There is no one-size-fits-all approach for interviewing – different groups prefer different formats. For example, people who use injection drugs were easiest to reach in person at harm reduction sites, while most other participants preferred scheduled Zoom or phone interviews. Flexibility helped us meet people where they were, literally and figuratively.

Despite the bumps along the way, we ultimately met our study goals by staying persistent, flexible, and community-centered. However, reaching LGBTQ+ participants in rural areas may take more time- and resource-intensive methods, such as traveling to recruit at in-person events or building relationships with trusted community gatekeepers. These methods were beyond our budget and timeframe for the current PrEP needs assessment; however, we now know to plan for more intensive outreach in future projects focused on rural LGBTQ+ populations.

What strategies have worked for you in reaching LGBTQ+ people in rural areas? We’d love to hear your thoughts in the comments!

The American Evaluation Association is hosting LGBTQ+ Voices in Evaluation TIG Week with our colleagues in the LGBTQ+ Voices in Evaluation Topical Interest Group. The contributions all this week to AEA365 come from our LGBTQ+ Voices in Evaluation TIG members. Do you have questions, concerns, kudos, or content to extend this AEA365 contribution? Please add them in the comments section for this post on the AEA365 webpage so that we may enrich our community of practice. Would you like to submit an AEA365 Tip? Please send a note of interest to AEA365@eval.org. AEA365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators. The views and opinions expressed on the AEA365 blog are solely those of the original authors and other contributors. These views and opinions do not necessarily represent those of the American Evaluation Association, and/or any/all contributors to this site.