Giving Compass' Take:

• While evidence-based policy is meant to improve lives, Straight Talk On Evidence points out that most social programs and practices don't produce the hoped-for results when rigorously evaluated.

• Should we rethink the methods behind studying policies? Not so fast. Even the failures and disappointments can teach us something. It's worth going through the process with the understanding that change is hard.

• While this is a good assessment and reality check for social innovators, we should also note exactly how evidence can be transformed into impact philanthropy, as Third Sector Capitol Partners details here.

Our main goal is to identify social programs and practices (“interventions”) that are backed by credible evidence of sizable, sustained effects on important life outcomes so that such interventions can be expanded in order to improve people’s lives on a larger scale. We summarize these interventions on the Social Programs That Work website and share the findings with government and foundation officials to help inform policy decisions.

Reviewing thousands of evaluation studies over the years has also given us a profound appreciation of how challenging it is to find interventions, such as those above, that produce a real improvement in people’s lives.

Across a range of human endeavors, most new interventions when rigorously evaluated are found to produce outcomes that are not meaningfully better than those of a control group, which receives (in most studies) usual services — such as standard medical treatment (in a medical RCT), usual business practice (in an RCT of a business strategy), or existing school practices (in an education RCT).

Examples from various disciplines include:

  • Business: Of 13,000 RCTs conducted by Google and Microsoft to evaluate new products or strategies in recent years, 80 to 90 percent have reportedly found no significant effects.
  • Medicine: Reviews in different fields of medicine have found that 50 to 80 percent of positive results in initial clinical studies are overturned in subsequent, more definitive RCTs. Thus, even in cases where initial studies — such as comparison-group designs or small RCTs — show promise, the findings usually do not hold up in more rigorous testing.
  • Education: Of the 90 educational interventions evaluated in RCTs commissioned by the Institute of Education Sciences and reporting findings between 2002 and 2013, close to 90 percent were found to produce weak or no positive effects.
  • Employment/training: In Department of Labor-commissioned RCTs that reported findings between 1992 and 2013, about 75 percent of tested interventions were found to have found weak or no positive effects.

The bottom line is that it is harder to make progress than commonly appreciated. While judgments may differ about the credibility of a particular study or the policy importance of a particular impact finding, the pattern of disappointing effects for most rigorously-evaluated programs — along with findings of important positive effects for a few — is compelling and transcends multiple fields. It needs to be taken seriously.

Read the full article about solving social problems when evaluations find disappointing effects from Straight Talk On Evidence.