Evaluating different strategies for estimating treatment effects in observational studies

ABSTRACT

Since the introduction of the propensity score (PS), methods for estimating treatment effects with observational data have received growing attention in the literature. Recent research has added substantially to the number of available statistical approaches for controlling confounding in such analyses. However, researchers need guidance to decide on the optimal analytic strategy for any given scenario. To address this gap, we conducted simulations evaluating both well-established methods (regression, PS weighting, stratification, and matching) and more recently proposed approaches (tree-based methods, local control, entropy balancing, genetic matching, prognostic scoring). The simulation scenarios included tree-based and smooth regression models as true data-generation mechanisms. We evaluated an extensive number of analysis strategies combining different treatment choices and outcome models. Key findings include 1) the lack of a single best strategy across all potential scenarios; 2) the importance of appropriately addressing interactions in the treatment choice model and/or outcome model; and 3) a tree-structured treatment choice model and a polynomial outcome model with second-order interactions performed well. One limitation to this initial assessment is the lack of heterogeneous simulation scenarios allowing treatment effects to vary by patient.

KEYWORDS:

• Average treatment effect
• counterfactuals
• matching
• measured confounding
• observational data
• propensity score
• regression
• tree
• weighting

Acknowledgments

The authors wish to acknowledge Dr. Yoko Tanaka for reviewing the manuscript and providing valuable comments and two anonymous referees whose comments led to a substantially improved paper.