Giving Compass' Take:

• Mary Beech explores the concept of external validity: When organizations examine other organizations' successes as guidance for programs.

• How strong are the measurement tools we use in general? How much can Randomized Controlled Trials (RTCs) make a difference in evaluating impact?

• Read more about the role of Randomized Control Trials in social progress.

When assessing program impact, researchers often focus on evaluating the causal relationship between two variables and the accuracy of those results. This concept is called “internal validity,” which asks the question: What’s the likelihood that this relationship actually exists? Another question equally important to policymakers is whether the observed results can be generalized to other settings, which is referred to as external validity. In other words, if we were to change characteristics such as time, setting, or population, would the causal relationship still exist?
There are a number of scenarios in which the external validity question is relevant. Take, for example, a county that conducted a rigorous five-year randomized control trial (RCT) to determine the impact of a family housing program. The evaluation found that the program improves adult and child well-being.

Based on this finding, the government wants to
  • Continue to fund the program for the next five years
  • Scale the program to new neighborhoods
  • Select additional service providers to deliver the same program model
  • Expand the program to serve homeless veterans in addition to families

As the social sector continues to identify and scale promising program models, we must think critically about how to best generalize and apply learnings. Exclusive use of RCTs is not scalable as programs expand. Continual monitoring, feedback, and learning builds on the prior experimentation and can enhance external validity as a program scales.

Read the full article on external validity by Mary Beech at Third Sector Capital Partners.