JMDE
Journal of MultiDisciplinary Evaluation
Number 3,
October 2005
ISSN 1556-8180
Editors
E. Jane Davidson &
Michael Scriven
Associate Editors
Chris L. S. Coryn &
Daniela C. Schröter
Assistant Editors
Thomaz Chianca
Nadini Persaud
Lori Wingate
Ryo Sasaki
Brandon W. Youker
Webmaster
Joe Fee
—The news and thinking
of
the profession and discipline of evaluation
in the world, for the world—
A peer-reviewed journal published in association with
The Interdisciplinary Doctoral Program in
Evaluation
The
Editorial Board
|
Katrina Bledsoe |
Shawn Kana'iaupuni |
|
Nicole Bowman |
Ana Carolina Letichevsky |
|
Robert Brinkerhoff |
Mel Mark |
|
Tina Christie |
Masafumi Nagao |
|
J. Bradley Cousins |
Michael Quinn Patton |
|
Lois-Ellen Datta |
Patricia Rogers |
|
Stewart Donaldson |
Nick Smith |
|
Gene Glass |
Robert Stake |
|
Richard Hake |
James Stronge |
|
John Hattie |
Dan Stufflebeam |
|
Rodney Hopson |
Helen Timperley |
|
Iraj Imam |
Bob Williams |
Table of Contents
PART III
American Journal of Evaluation
Evaluation: The International Journal of Theory, Research
and Practice
Volume 26, number 1 was the first issue of the American Journal of Evaluation (Volume 26, Number) with Robin Miller at the helm as Editor. There are 5 articles, plus 1 contribution each in “Forum,” “Methods Notes,” “The Historical Record,” and “Ethical Challenges.”
In the issue’s first article, “An Alternative Route to Policy Influence,” Carol Hirschon Weiss, Erin Murphy-Graham, and Sarah Birkeland identify a type of evaluation influence based on their study of the use of D.A.R.E. evaluations. They posit that this type of influence (or “route to influence”), which they call “imposed use,” is distinct from those commonly discussed in the literature on evaluation use, namely instrumental, conceptual, and political/symbolic. “Imposed use,” they write, “may occur in any field where a higher level of government with funds to disburse demands specific action on lower operation levels, based on evidence” (p. 25). In the case of D.A.R.E., the U.S. Department of Education made instrumental use of evaluation findings by limiting funding to programs that met certain criteria (based on evaluation results). School districts’ response to this mandate (which in most cases was to drop their D.A.R.E. programs) exemplifies imposed use. That is, they were using evaluation findings to make decisions—in fact the authors report the districts had little interest in the evidence. Rather, they were reacting to the federal requirement based on those findings. Weiss, Murphy-Graham, and Birkeland note “imposed use is not something new . . . it is a concept that has not surfaced before in the evaluation literature” (p. 25).
In “Quality, Context, and Use: Issues in Achieving the Goals of Metaevlauation,” Leslie Cooksy and Valerie Caracelli report on their metaevaluation of 87 evaluation reports prepared by International Agricultural Research Centers. They highlight how common evaluation issues—particularly quality criteria, political and cultural context, and use—have a “different texture” (p. 40) in metaevaluation contexts. It is crucial to clearly identify the purpose of a metaevaluation—whether for evaluating evaluation quality or identifying evaluations for inclusion in an evaluation synthesis—so that appropriate planning and methodological decisions can be made. The authors use examples from their metaevaluation experience to illustrate these issues.
Laurie Stevahn, Jean King, Gail Ghere, and Jane Minnema’s article on “Establishing Essential Competencies for Program Evaluations” present a detailed taxonomy of “essential competencies for program evaluators.” They argue that improved training, enhanced reflective practice, advanced research on evaluation, and professionalization of the field are benefits that are likely to result from acceptance and use of such a taxonomy. They identify 61 competencies across 6 domains: professional practice, systematic inquiry, situational analysis, project management, reflective practice, and interpersonal competence. Furthermore, they crosscheck these competencies again the Joint Committee standards, AEA Guiding Principles, and Canadian Evaluation Society Essential Skills Series. Stevahn and her colleagues conclude by identifying two activities that should be undertaken to validate the taxonomy for widespread use: a comprehensive validation study and the construction of rubrics for each competency that specify proficiency levels.
In this issue’s fourth article, Melanie Ehren, Frans Leeuw, and Jaap Scheerens demonstrate the use of a “policy scientific approach” to reconstruct the program theory of the Dutch Edcuational Supervision Act. The three steps of this approach are to identify assumptions, validate the reconstruction, and critically evaluate the program theory. After presenting the results of their study, they discuss the benefits of limitations of the approach.
In the final article, ““Multidimensional Implementation Evaluation of a Residential Treatment Program for Adolescent Substance Abuse,” Leyla Faw, Aaron Hogue, and Howard Liddle discuss the importance of evaluating a program’s implementation and report on their evaluation of the structure and process of a substance abuse treatment program. They conclude that “understanding the effectiveness of treatment for adolescents hinges on the continued development of methods to measure treatment implementation and analyzing these findings in relation to outcomes” (p. 93).
In the Forum section of this issue, Thomas Schwandt discusses “The Centrality of Practice to Evaluation.” He analyzes the popular conception of “evidence-based,” which he says values evidence over practice, and discusses what implications this view of “evidence-based” has for understanding practice and evaluation. He recommends that evaluators move from thinking of practice as “an objective that needs to be repaired” to a more genuine conceptualization in which practice is a “material and linguistic event in which human dilemmas emerge and are addressed” (p. 100).
In the Methods Notes Section, J. Jackson Barnette and Anne Baber Wallis seek to “close one of the few gaps left in the Campebll-Stanley-Cook-Shadish legacy of research designs” (p. 106). They examine how what happens to an intervention between multiple postobservations (e.g., removal, continuation, changes in intensity) in experimental and quasi-experimental evaluation designs impacts validity, data modeling, and analysis. They argue that designs that take these factors into account will produce better inferences.
Donna Mertens’ contribution in
the “Historical Record” section provides an account of the “The Inauguration fo
the International Organization for Cooperation in Evaluation” (IOCE). IOCE’s mission
is “to help legitimate and strengthen evaluation societies, associations or
networks so that they can better contribute to good governance, effective
decision making, and strengthen the role of civil society” (p. 127). Mertens
describes the work done to get the organization off the ground; gives a brief
account of the inaugural assembly that took place in
In the “Ethical Challenges” section, Gillian Kerr comments on two analyses of “The Steering Committee” ethical challenge in a previous issue of AJE. She did not think these analyses paid sufficient attention to “the role of the steering or advisory committee itself and the extent to which membership of such a committee is associated with genuine power” (p. 132) and explains why in “Reflections of ‘The Steering Committee.’”
The Spring 2005 issue of New Directions for Evaluation, Teaching Evaluation Using the Case Method, edited by Michael Q. Patton and Patricia Patrizi is intended to advance the practice of evaluation teaching using the case method by “providing specially developed cases for teaching and teaching guidelines and discussion points to use in conjunction with the cases” (p. 3). In this issue, chapters 2-4 conclude with “Teaching Guidelines and Questions,” which are intended to provide general case teaching guidance by providing case teaching questions and evaluation points to elicit through questioning.
Chapter 1, Case Teaching and Evaluation, by Michael Q. Patton and Patricia Patrizi, outlines the logic and likely benefits of using and applying cases as a teaching method for students of evaluation. The authors argue that case teaching and training, like the longstanding traditions of using cases for teaching law and medicine, will prepare future evaluators for the practical problems that arise in real-world evaluations (e.g., “professional practice does note lend itself to rules and formulas” and “decisions are rarely routine”, p. 5). The strategies for case teaching strategies presented by the authors in this chapter include (1) facilitating case discussion to provide experiences in evaluative thinking, situational analysis, and practical problem solving for real-world evaluation, (2) set and model norms of civil interaction, (3) emphasizing advanced preparation, (3) setting expectations and creating a learning frame of mind, (4) starting the questioning process by eliciting the facts of the case, (5) vive la difference [e.g., reconciling opposing points of view], (6) adding hypothetical and incorporating role playing, (7) concluding with takeaways and generalized learning, and (8) supporting active, practice-oriented learning. Patton and Patrizi conclude the chapter by stating that
Evaluation as a field of professional
practice has long way to go to achieve the prestige of fields like law,
medicine, and business, but the challenges we face in supporting the
development of skilled practitioners who can analyze unique situations, deal
with diverse people, and exercise astute judgment bear striking similarities to
these professions.
(p. 13)
In Chapter 2, Evaluation of the Fighting Back Initiative, by Kay E. Sherwood, presents the case of the Robert Woods Johnson Foundation’s Fighting Back initiative, an $88 million dollar investment by the foundation for developing community-generated strategies for reducing use and abuse of alcohol and illegal drugs. This investment included $14 million for an independent evaluation of the foundation’s initiative. In the case, Sherwood provides all of the necessary background and contextual information for making the case a usable teaching tool. Also presented in the case are early efforts at evaluating the initiative, beginning in 1990, where the evaluation floundered as the research team was “unable to manage the complexity and comprehensiveness of the design” (p. 23). This team purportedly wasted $4.6 million, 4 years, baseline for future efforts, and credibility for the overall effort. Eventually the evaluation was rescued by a new research team, which conducted the 1994-2000 evaluation of the initiative. All in all, the case of the Fighting Back Initiative provides a rich, complex teaching example.
In Chapter 3, Evaluation of the Central Valley Partnership of the James Irvine Foundation, by Martha S. Campbell, Michael Q. Patton, and Patricia Patrizi, the case presented was initiated by the foundation as a “partnership for citizenship” (p. 39). Thus, the purpose of the Central Valley Partnership (CVP) was to engage low income, immigrant, and disenfranchised residents in civic action. In this example, the authors present a case where the role of the evaluator shifts from pure evaluation to “an organizational development resource” (p. 46). In this sense, the case illustrates the various roles and responsibilities that evaluators are often required or requested to perform. The case concludes with comments from Martha Campbell, now the vice president for programs at the Irvine Foundation, in which she states
(p. 54)
Chapter 4, Evaluating Home Visitation: A Case Study of Evaluation at the David and Lucile Packard Foundation, by Kay E. Sherwood, presents a case where the foundation used an evaluation-focused strategy to making grants for child development projects. Through this strategy, the foundation’s evaluation efforts frequently emphasized results-based evidence to support project effectiveness, primarily in the form of experimental designs. Unfortunately, as the case presents, these effects were generally “mixed” or “non-significant” (p. 67). Much of the case involves the publication of these poor, disappointing results and the subsequent fallout generated by them, including efforts for damage control by the foundation and other stakeholders.
In Chapter 5, Evaluation Case Teaching from a Participant Perspective, by John Bare, the author describes the benefits of the case teaching method from the view of a learner. Most interesting in Bare’s chapter is the “surfacing of values,” wherein the author argues that values are pervasive and shape both program planning and evaluation. Moreover, the author notes that “cases help reveal these” (p. 89).
The issue concludes with Chapter 6, Diverse and Creative Uses of Cases for Teaching, by Michael Q. Patton. In this chapter Patton presents suggestions for using the cases presented in the issue, and other cases, for the “broader context of evaluation teaching and training” (p. 91). First, the author provides issues for exploring cross-case comparisons including (1) connecting parts into a whole, (2) the personal factor, (3) evaluator roles and purposes, (4) complex relationships and institutional arrangements, (5) controversies and politics, and (6) what is missing? Second, Patton explores additional teaching uses for cases. These uses could include (1) insights into evaluator competencies, (2) learning to write executive summaries, (3) practicing qualitative analysis and extracting lessons learned, (4) stakeholder analysis and stakeholder mapping, (5) developing ethical commitments and sensitivities, (6) metaevaluation training, and (7) applying model, theorists, and conceptual distinctions. Patton summarizes the issue by stating that
This volume on using cases for
teaching evaluation aspires to contribute to professional excellence in
evaluation by grounding training real-world experiences captured and presented
in detailed cases. Case teaching and the additional practice-oriented teaching
ideas presented in this chapter seek to bridge the gap between knowing and
doing.
(p. 98)
As a student of evaluation I found “Teaching Evaluation Using the Case Method” a compelling, logical approach to teaching and learning evaluation. Each of the cases presented in Chapters 2-4 offer a unique series of problems and possibilities. Furthermore, I found Patton’s presentations of teaching guidelines and questions at the end of these chapters useful and relevant to the cases presented. While I agree with Patton that evaluation teaching and training needs to “bridge the gap between knowing and doing” (p. 98), there are alternatives to cases which should be considered as well. For example, cases may in fact be “real-world,” but the use of the case is still “hypothetical.” That is, learners are not really evaluating the programs or projects presented in the cases. They may be confronted with the complexities and problems of real-world evaluation, but real-world practice should include “real” evaluation as opposed to merely practicing on cases. Although cases are an invaluable teaching tool, I would argue that what many professional programs of study call “field or professional experience” would be the real, real-world equivalent of cases.
References
Patton,
M. Q. & Patrizi, P. (Eds.) (2005). Teaching evaluation using the case
method. New Directions for Evaluation,
105.
In a time of results-based management and
budgeting, the question whether or not the inputs
have been in line with the policies of donors and partner countries is not
longer really relevant. The real question is whether the results of our actions are in line with the policies and the
problems that these policies tried to address.
(van den Berg, p. 35)
The
first 2005 issue of Evaluation (Volume
11(1), January 2005) begins with two contributions to A Visit to the World of Practice, both
of which focus on results-based evaluation and impact assessment within the
context of the Millennium Development Goals (MDGs). Please visit http://ddp-ext.worldbank.org/ext/MDG/-home.do
for information on the MDGs.
First,
Kusek, Rist, and White discuss how the shift from implementation-focused
monitoring and evaluation (M&E) to results-based M&E is taking place in
various developed and developing countries, which challenges are being faced in
this transition, and what strategies should be considered when introducing
results-based M&E, including readiness assessments, political and
organizational issues, and potential challenges with implementation, reliable
data collection and analysis.
Second,
van den Berg discusses some methodological issues in the assessment of
development cooperation. Monitoring, for example, would not assess if the right
things are done in development, but only whether things are done right. Impact
assessments, in contrast to monitoring, would be complicated and expensive,
because impacts occurs over long terms, require increased scope of research,
and rely on baseline data often unavailable. Moreover, counterfactuals have to
be considered to indicate that observed outcomes in fact resulted from the
intervention under investigation. Causality as the key to the establishment of
impact would be reduced in the social science context to “specific causality”, because
there are no general laws as in the natural sciences. To proof linkages between
outcomes and impact, methods such as lab research, RCTs, and double-blind
studies with comparison groups are commonly utilized by social scientist. Van
den Berg argues for the methodological inclusion of historical analysis to
ascertain causality, utilizing triangulation “par excellence” to insure
reliability and reasoning for validity. Current evaluation practice employs
triangulation only methodologically. However, using historical triangulation
eliminates the need for counterfactuals to establish causality. Moreover,
linear causality as established through statistical techniques is often
thwarted by societal complexities. Therefore, discussions in social sciences
should shift toward “conditionalities” (p. 34). Van den Berg believes “that the
development community should move from causality or plausibility to
contribution, and from direct linkages to necessary but not sufficient
conditions for change” (p. 34).
Four
articles follow. First, Saunders, Charlier, and Bonamy discuss how evaluation
can be used to support change, exemplified in two international higher
education case examples. Second, Kautto and Similä provide an account of
evaluating “recently introduced policy instruments (RIPIs)” (p. 55) supported
by intervention theories and recommend (1) the utilization of theory-based
approaches, (2) the selection of criteria and establishment of causal links
between evaluation criteria, (3) the selection of causal linkages for which
information can readily be ascertained, (4) determination of procedures for
proceeding with the criteria for which information is not readily available,
and (5) consideration of potential for theory failure. Third, Byng, Norman, and
Redfern provide a case example within a mental health context, utilizing
realistic evaluation as coined by Pawson and Tilley in combination with
analytic induction. Fourth, Shadish, Chacón-Moscoso, and Sánchez-Meca describe
how meta-analysis and systematic reviews have been developed historically,
utilized in
In
the Review section of Evaluation 11(1), Kushner looks at a
current UK Cabinet Publication entitled “Quality in qualitative evaluation: a
framework for assessing research and evidence.”
The
final section, News from the Community,
discusses the fifth annual Japanese Evaluation Society (JES) and third annual
African Evaluation Association conferences. The section also introduces the
International Organization for Cooperation in Evaluation (IOCE; also see this
issue of JMDE). The final news from the community is the Univation/German
Evaluation Society conference, which focused on network evaluation.