Giving Compass' Take:

• Jill Barshay explains how researchers have identified fraud and widespread bias in 30 years of education research that has been used to shape schools. 

• How can funders rely on evidence-based interventions in the face of biased research? 

• Learn about improving research transparency and replication. 

An analysis of 30 years of educational research by scholars at Johns Hopkins University found that when a maker of an educational intervention conducted its own research or paid someone to do the research, the results commonly showed greater benefits for students than when the research was independent. On average, the developer research showed benefits — usually improvements in test scores —  that were 70 percent greater than what independent studies found.

“I think there are some cases of fraud, but I wouldn’t say it’s fraud across the board,” said Rebecca Wolf, an assistant professor in the Center for Research and Reform in Education at Johns Hopkins University and lead author of the draft study.  “Developers are proud of their products. They believe in them. They’ve worked hard in developing these products. They want a study that puts the best face forward.”

Biased research matters because current federal law encourages schools to buy products that are backed by science. In order to tap into federal school improvement funds, for example, low-achieving schools with disadvantaged children are required to select programs that have been rigorously tested and show positive effects.

The study, “Do Developer-Commissioned Evaluations Inflate Effect Sizes?” was presented at a March 2019 conference session of the Society for Research on Educational Effectiveness (SREE) in Washington, D.C.  The paper is a working paper, meaning it has not yet been published in a peer-reviewed journal and may still be revised.

Read the full article about bias in education research by Jill Barshay at The Hechinger Report.