Giving Compass' Take:

• This Social Innovation Research Center post explores the challenges in research replication, what that means for evidence-based policies/programs and how we can address them.

• From pre-registered research plans to sharing data, the evaluators at MDRC has thought through this "crisis" and offered solutions. Do we have the resources to put them into practice?

• Here's how psychology’s replication crisis has made the field better.

One of the central tenets of evidence-based policy is that programs and policies that are deemed “evidence-based” should be replicable in other settings. This is not always easy — even interventions backed by solid evidence can be hard to replicate. There are lots of potential reasons for this, including differences in the target populations, different settings, poor implementation, and insufficient fidelity to the original model, to name a few.

The research replication crisis is a problem, not just for basic science, but for the social sciences, too. For example, one recent attempt to repeat the findings of 100 peer-reviewed psychology studies found that only 39 could be reproduced. Empirical economics has faced its own similar replication crisis. Stanford professor John Ioannidis, a long-time critic of research more generally, has argued that the primary reasons are poorly-designed studies and researcher bias.

 

What Can Be Done?

Enter MDRC, one of the nation’s best evaluators. One of their researchers, Rachel Rosen, wrote an interesting and somewhat timely blog post earlier this month that describes what her organization is doing to deal with this.

Their efforts are worth a look. They include:

  • Pre-registering Research Plans
  • Requiring More than Statistical Significance
  • Sharing Data
  • Conducting Replication Studies

Addressing the replication crisis will be vital if the evidence-based policy movement is to be successful. Continued investments in clearinghouses and registries are one important strategy. But so too are basic best practices for researchers.

Read the full article about ways to address the evidence based policy crisis by Patrick Lester at Social Innovation Research Center.