Sample Selection in Randomized Experiments: A New Method Using Propensity Score Stratified Sampling

Abstract

Randomized experiments are often seen as the “gold standard” for causal research. Despite the fact that experiments use random assignment to treatment conditions, units are seldom selected into the experiment using probability sampling. Very little research on experimental design has focused on how to make generalizations to well-defined populations or on how units should be selected into an experiment to facilitate generalization. This article addresses the problem of sample selection in experiments by providing a method for selecting the sample so that the population and sample are similar in composition. The method begins by requiring that the inference population and eligibility criteria for the study are well defined before study recruitment begins. When the inference population and population of eligible units differs, the article provides a method for sample recruitment based on stratified selection on a propensity score. The article situates the problem within the example of how to select districts for two scale-up experiments currently in recruitment.

Keywords:

• Experimental design
• scale-up
• sampling
• stratified
• recruitment

View correction statement:

Corrigendum

Notes

Although for proprietary reasons we do not include the actual numbers of districts purchasing these programs from MGH, in the actual analyses these numbers were used.