Comparing the performance of statistical methods that generalize effect estimates from randomized controlled trials to much larger target populations

Abstract

Policymakers use results from randomized controlled trials to inform decisions about whether to implement treatments in target populations. Various methods—including inverse probability weighting, outcome modeling, and Targeted Maximum Likelihood Estimation—that use baseline data available in both the trial and target population have been proposed to generalize the trial treatment effect estimate to the target population. Often the target population is significantly larger than the trial sample, which can cause estimation challenges. We conduct simulations to compare the performance of these methods in this setting. We vary the size of the target population, the proportion of the target population selected into the trial, and the complexity of the true selection and outcome models. All methods performed poorly when the trial size was only 2% of the target population size or the target population included only 1,000 units. When the target population or the proportion of units selected into the trial was larger, some methods, such as outcome modeling using Bayesian Additive Regression Trees, performed well. We caution against generalizing using these existing approaches when the target population is much larger than the trial sample and advocate future research strives to improve methods for generalizing to large target populations.

Keywords:

• Causal inference
• External validity
• Generalizability

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

The research reported here was supported in part by the Institute of Education Sciences, U.S. Department of Education, through grant R305D15003 (PIs: Stuart and Olsen), the National Institute of Mental Health, through grant T32MH10943602 (PIs: Barry and Stuart), and the National Institute on Drug Abuse, through grant R01DA036520 (PI Mojtabai) and grant R00DA042127 (PI Rudolph).