Naaima Khan

Founder and Principal Consultant

Create Good

Decolonizing Evaluation: Navigating Contradictions

Evaluation is often framed as a neutral technical exercise, yet it has functioned as a technology of colonization that reproduces violence and erasure by determining whose knowledge counts and who gets to define "success." This session navigates the contradiction between mandates for institutional longevity and the community-led imperative for self-determination. We will examine case studies from grassroots organizations navigating these power dynamics, highlighting how traditional metrics can obscure collective outcomes or misrepresent community priorities. Finally, we will introduce a few strategies for beginning to shift this paradigm, including decolonizing approaches to planning and design, and understanding what success looks like.

A word of caution: thirty minutes cannot provide a holistic framework or checklist for "decolonize" anything, and this session won't pretend to. Decolonization entails redistribution of power, resources, and authorship, led by communities rather than practitioners. What we can do in the time we have is look closely at a few places where colonized logic hides in ordinary evaluation practice. We will follow the money and the language, asking what solidarity looks like when it involves invitation and shared risk. You will leave with better questions, a clearer view of your own seat in the arrangement, and a sense of where the longer work starts.

Melanie Hwalek, Ph.D.

CEO, SPEC Associates

Assistant Professor, Michigan State University

Using Retrospective Pre Tests: Design Matters

The Retrospective Pre Test (RPT) method has been used since 1947 and is one of the quasi-experimental designs in Campbell and Stanley’s (1963) seminal book Experimental and Quasi-Experimental Designs in Research. It has been the subject of numerous articles debating its validity for measuring program outcomes. Melanie Hwalek has been intrigued with RPT for the past 25 years and has amassed a library of published literature on the topic. Recently, Melanie and her colleagues at SPEC Associates and the Alzheimer’s Association Michigan Chapter studied whether different formats for displaying RPT items on a survey questionnaire affect the quality of data obtained and outcome results. In this presentation, Melanie will provide an overview of the controversies surrounding RPT, describe the criteria used for assessing survey layout quality, and share the study results. Participants will have an opportunity to ask questions, challenge the study’s conclusions, think about the importance of paying attention to survey data quality, and discuss ways the study might be replicated in other contexts. 

Fernando Ospina, Ph.D.

Assistant Professor

Department of Biomedical Sciences Population Health Research

Western Michigan University Homer Stryker M.D. School of Medicine

Champions, Challenges, and Change: Data Use to Reduce Black Infant Mortality

Black Infant Mortality disparity has been a persistent problem in Kalamazoo for decades. While previous efforts to reduce Black infant mortality have undergone fits and starts, Cradle Kalamazoo has been able to sustain attention to Black infant mortality for over a decade. This collective impact efforts has mobilized individuals and organizations throughout the community, generating action including: raising awareness about the role of racism in infant mortality, educational programming to reduce safe sleep deaths, creating/sustaining programs and services to support birthing families, and prompting multiple evaluation efforts and research projects, informed by stakeholders. 

Evaluation and data, embedded into an agenda for research and community improvement, has played a prominent role in driving and directing local maternal and infant health improvement efforts. This presentation recounts the role of data in the formation of Cradle Kalamazoo and its ongoing efforts. The presenter will discuss the role of key champions who used data to mobilize others; challenges evaluating collective impact; and provide examples of instrumental and conceptual data uses. The presentation will conclude with future directions for an evaluation and research agenda to improve maternal and infant health Kalamazoo and how it builds upon previous efforts.

Nate Mandel

Senior Fellow

Stand Together Trust

Net Promoter Score in Nonprofit Evaluation: One Signal in an Emerging Practice of Customer Listening

NPS has become the most widely used client-feedback question in the nonprofit sector — despite being built in a for-profit, customer-loyalty context that doesn't map cleanly onto mission-driven work. Drawing on a decade of building nonprofit feedback systems, first at a workforce nonprofit and then across national philanthropic networks, this session traces NPS from an individual case-management tool to a field-wide practice, showing how a single low score can prevent a participant from dropping out, how aggregated scores can surface early operational problems, and what happens when funders, evaluators, and nonprofits start treating client feedback as a genuine signal that can guide decision making. We'll cover courtesy bias, segmentation, and where the field is headed as NPS becomes one part of a broader customer-listening movement.

Courtney Stone

Assistant Professor

University of Illinois Urbana-Champaign

From Values to Validity: Operationalizing Culturally Responsive Evaluation Through Multicultural Validity

Culturally responsive evaluation (CRE) challenges evaluators to attend to values, systems of power, histories, and lived experiences of diverse communities. Yet, instituting these commitments into evaluation practice remains a challenge. This presentation situates Kirkhart’s theory of multicultural validity as a technical framework for operationalizing many of the core commitments of CRE. The theory roots validity arguments and evidence in culture through the inclusion of diverse epistemological perspectives, theoretical frameworks, and methods. Using a longitudinal STEM program evaluation as an anchor example, alongside illustrations from the evaluation literature, this presentation will explore how relational, experiential, methodological, theoretical, and consequential evidence can inform decisions throughout the evaluation process—from defining evaluation questions and engaging interest holders to interpreting findings and supporting use. Rather than presenting multicultural validity as a separate validity framework, this session positions it as a means of translating the values of culturally responsive evaluation into defensible evaluation practice.