WMU researcher spends fellowship period on education research registry
KALAMAZOO, Mich.—A Western Michigan University education researcher has spent the 2015-16 academic year in a fellowship post focused on putting together a registry that could vastly improve the structure and transparency of impact studies in education.
Dr. Jessaca Spybrook, associate professor of educational leadership, research and technology, is completing a fellowship with the Society for Research on Educational Effectiveness. She led a registry project that will improve the structure and availability of information on impact studies in education. The registry, which will be pilot tested this summer, would not only inform those undertaking studies to help improve study design, but also make study details accessible to funders, policy-makers and other researchers.
"I'm so excited about this," Spybrook says. "I feel like this is very timely. There are a lot of issues across the social sciences in general with transparency and replication."
About the project
The Society for Research on Educational Effectiveness was founded some 10 years ago with the goal of promoting rigorous research in education. The society recently approached the Institute of Education Sciences, a division of the U.S. Department of Education, and obtained just under $1 million in funding, part of which is dedicated to creating a registry that would improve impact studies and make their findings more transparent. The society then granted Spybrook a fellowship to put the registry together, working closely with Dr. Rebecca Maynard, University Trustee Professor of Education and Social Policy at the University of Pennsylvania, and Dr. Larry Hedges, Board of Trustees Professor of Statistics and Education and Social Policy at Northwestern University, the project's principal investigators.
Calls for improving transparency across the social sciences are increasing, Spybrook says. Two obstacles confronting research efforts are referred to as the "file drawer" problem and "fishing."
Too often, studies that produced null or negative findings go unreported, while studies showing positive results get more publicity. Those not showing positive results are stuffed in the "file drawer" and go unpublished. The null findings from well-designed studies, however, are as valuable as affirmative findings for arriving at conclusions about the effectiveness, or lack thereof, of particular programs, policies or practices.
Fishing refers to the selectivity of reported findings. Researchers report outcomes that produce estimates of impacts that are statistically significant. The reported findings often appear more favorable than they would have had all the findings been reported.
What it will do
Creating a registry of studies would help ensure all findings of all studies are reported and would also improve the rigor of studies as they are being conducted, Spybrook says.
"What will happen is, after you fill out a pre-registry, you'll be given a registration number," Spybrook explains. "It will do two things: One, it will help researchers think about all facets of their study upfront. And, two, it will put that accountability piece into play."
Researchers also can check the registry to see what other studies are being conducted and what already is being undertaken. The registry also will put that information at the fingertips of study funders and policy-makers.
"We're not using this to rate your study," Spybrook says. "We're not using this to judge the quality of the proposal. In essence, it's going to be a service to the field to increase the capacity of people to do these studies, build accountability and transparency and reduce some of that publication bias."
Ultimately, the registry could improve the structure and transparency of impact studies and make sure findings, both negative and positive, are reported.
"We want to assemble a huge database," Spybrook says.
Rollout of the registry is planned for late fall.
Spybrook and her colleagues presented the project in March at the Society for Research on Educational Effectiveness Spring Conference in Washington, D.C. The project was well-received by conference attendees.
"We had a very positive response," Spybrook says. "People could really see the value of this."
For more news, arts and events, visit wmich.edu/news.