Julien B. Kouame

Doctoral Dissertation Announcement

---

Candidate: Julien B. Kouame

Degree of: Doctor of Philosophy

Department: Educational Leadership, Research and Technology

Title: Design in Evaluation: Adequacy and Validity of Health Evaluation in the Context of Developing Countries

Committee:
Dr. Brooks Applegate, Chair
Dr. Marianne DiPierro
Dr. Michael Bamberger

Date: Wednesday, November 7, 2012 9 a.m. to 11 a.m.
Walwood Commons

Abstract:
The ramifications of poorly conducted international health intervention evaluations (IHIE) are that the health of recipients of the intervention may be placed at risk (Stoto & Cosler, 2005). Moreover, very few IHIEs meet globally established minimum criteria for sound methodology (Forssman, Gupta, and Burgess, 2006; Tones, Tilford, and Robinson, 1990).  This research study investigates the level of rigor in a sample of IHIEs, as well as delineates the institutional policies and procedures that govern the evaluations.  Specifically, this research seeks to answer four questions: (1) What are policies, guidelines, and requirements for program evaluation and evaluation reports, posed by international donors for evaluators?  (2) What are the common types of research designs used to evaluate international health interventions? (3) What are common components and contents of reports from evaluations of international health interventions? (4) What is the level of rigor of those designs use to evaluate international health interventions?

A sample of 55 online evaluation reports published between 2005 and 2010 representing seven organizations funding IHIEs are reviewed. Checklists developed by Miron (2004) Evaluation Report Checklist and the Checklist for Assessing Threats to Evaluation Validity developed by Bamberger, et., al. (2012) for reviewing mixed methods evaluations are used to quantify the adequacy of the sample of reports.

Findings show there is considerable variability and flexibility among the seven organizational evaluation policies and guidelines governing the evaluation of IHIEs.  Checklist analysis of the actual evaluation reports reveals that very few of the interventions use rigorous evaluation designs to address program impact. Furthermore, the evaluation reports, as written, only reflect the extent of the information/component parts required for reporting as stipulated by their specific policies and guidelines.  Unfortunately, this information is often not adequate to assess if purpose of the intervention has been successfully achieved, thus compromising the transparency of the evaluation report.

While this study examines a limited set of evaluation reports from IHIEs the implications of this research suggest that international funding bodies need explicit policies and procedures that guide both program evaluation design and evaluation reporting. Second, greater attention to both evaluation design and the components of the written evaluation report are needed to adequately represent program impacts.  Thirdly, coupling more rigorous evaluation designs that can fully address program impact with more systematic and comprehensive reporting will result in greater transparency, an element that all IHIPs funders expose.