Evidence-Based Practice

The Evaluation Center at Western Michigan University contracted with this project.

Title: Critical Review of Evidence-Based Program Repositories and Registers for Behavioral Health Treatment
Sponsor: U.S. Department of Health and Human Services | National Institutes of Health
Staff: Dr. Stephen Magura, principal investigator, Dr. Daniela Schroeter, co-principal investigator, and Dr. Chris Coryn, co-principal investigator.
Dates: July 1, 2011, through June 30, 2014
Contract: NIH No. 1R21DA032151-01

Description

Identifying what works in the behavioral health disciplines is crucial for making well-informed defensible decisions when selecting and implementing behavioral health-related interventions. As the need for relevant, accessible, and systematically derived information that supports policy decisions have increased, so has the demand for comprehensive sources of information about effective programs and intervention modalities. To this end, a number of evidence-based repositories and registers have been developed by various federal and state organizations, non-government organizations, and universities.

Unfortunately, the proliferation of potentially useful repositories and registers has presented policymakers and practitioners with ambiguous, inconsistent, and incomplete recommendations for putative evidence-based interventions. These databases often have dramatically different purposes, criteria for inclusion, definitions of acceptable evidence, and standards for designating interventions as effective. In order to provide policymakers and practitioners with information on how best to use evidence-based repositories and registers in the decision-making process, this study comprehensively and critically examines the strengths, weaknesses, and limitations of these evidence-based repositories and registers.

Project aims

1. Compile a comprehensive list of evidence-based program repositories and registers for behavioral health-related interventions.

2. Classify those EBPRS according to their purposes, methodologies, acceptable types and standards of evidence, and other factors that are used to include and certify effective interventions.

3. Determine the practical consequences of using different types and standards of evidence for including and certifying interventions as effective.

Potential study outcomes

This study sought to accomplish four main goals:

  1. Gain an understanding of assumptions about what constitutes sufficient empirical evidence related to what works in the behavioral health disciplines.
  2. Identify the varying definitions of program efficacy, effectiveness, and readiness for dissemination used in evidence-based practice.
  3. Understand and decompose the standards used to judge the absolute or relative effectiveness of the constituent interventions and modalities of intervention contained within evidence-based repositories.
  4. Understand the criteria used to assess the credibility of the evaluation or research studies that support inclusion in evidence-based repositories.

Progress and lessons learned

All phases of the study have been completed. Manuscripts will be submitted to the Journal of Evaluation and Program Planning, edited by Jonny Morrell.

Several key findings have been identified:

  • Maintaining current reviews tended to be a time and resource challenge cited by most of the registers.
  • Most registers tended to use word-of-mouth marketing and could use a broader dissemination strategy.
  • The evidence-based registers we have found varied somewhat in their structure, funding sources, content areas, and purposes.
  • The registers tended to agree that a Campbellian hierarchy (Gugiu, in press) was the best way to frame evidence. This type of model privileges evidence developed with well-controlled RCTs over evidence derived from other means.
  • The registers tended to use the Campbellian approach, however, they often differed in the ways they applied criteria for evidence quality, leading to discrepancies in program ratings across registers.
  • Typically the registers focused on several key user groups: decision-makers, practitioners, and researchers.

Article:

Means, S.N., Magura, S., Burkhardt, J.T., Schröter, D.C., Coryn, C.L.S. (2014) Comparing Rating Paradigms for Evidence-Based Practice Registers in Behavioral Health. Journal of Evaluation and Program Planning. Manuscript accepted for publication.

Schröter, D.C., Burkhardt, J.T., Magura, S., Means, S.N., Coryn, C.L.S. (2014). A Comparison of Evidence-Based Registers for Behavioral Health. Journal of Evaluation and Program Planning. Manuscript accepted for publication.