Improving Effect Estimates by Limiting the Variability in Inverse Propensity Score Weights

Abstract

This study describes a novel method to reweight a comparison group used for causal inference, so the group is similar to a treatment group on observable characteristics yet avoids highly variable weights that would limit statistical power. The proposed method generalizes the covariate-balancing propensity score (CBPS) methodology developed by Imai and Ratkovic (2014) to enable researchers to effectively prespecify the variance (or higher-order moments) of the matching weight distribution. This lets researchers choose among alternative sets of matching weights, some of which produce better balance and others of which yield higher statistical power. We demonstrate using simulations that our penalized CBPS approach can improve effect estimates over those from other established propensity score estimation approaches, producing lower mean squared error. We discuss applications where the method or extensions of it are especially likely to improve effect estimates and we provide an empirical example from the evaluation of Comprehensive Primary Care Plus, a U.S. health care model that aims to strengthen primary care across roughly 3000 practices. Programming code is available to implement the method in Stata.

Keywords:

• Causal inference
• Covariate balance
• Observational studies
• Power

Acknowledgments

The authors acknowledge the contributions of Stacy Dale, Timothy Day, Jonathan Gellar, Heather Ingraham, Huihua Lu, Sean Orzol, Debbie Peikes, Liz Potamites, Pragya Singh, and a large team at Mathematica who collectively prepared the data for the empirical application.

Notes

1 Notes

Kang and Schafer (2007, p. 529) and Imai and Ratkovic (2014, p. 253) demonstrated that various estimators perform well when the researcher has access to the ${\mathbf{v}}_i$; therefore, we focus on scenarios where the ${\mathbf{v}}_i$ are unavailable.

2 Weights from the logistic regression model were trimmed to 0.10 or 0.90 if they fell outsize the range [0.10, 0.90]; 0.10 and 0.90 are the cutoffs suggested by Crump et al. (2009). The random forest classifier (RF) included 100 trees, with tree depths no greater than 5, leaf sizes no smaller than 20, and all 4 covariates. These parameters were chosen through a parameter grid search and 10-fold cross-validation.

Additional information

Funding

This work was funded under Mathematica’s independent evaluation of the CPC + model for the U.S. Department of Health & Human Services, Centers for Medicare & Medicaid Services, Center for Medicare & Medicaid Innovation (contract HHSM-500-2014-00034I/HHSM-500-T0010).