Covariate Balance for Observational Effectiveness Studies: A Comparison of Matching and Weighting

References

• Austin, P. C. (2009a). Balance diagnostics for comparing the distribution of baseline covariates between treatment groups in propensity-score matched samples. Statistics in Medicine, 28(25), 3083–3107. https://doi.org/10.1002/sim.3697
   PubMed Web of Science ®Google Scholar
• Austin, P. C. (2009b). Some methods of propensity-score matching had superior performance to others: Results of an empirical investigation and Monte Carlo simulations. Biometrical Journal. Biometrische Zeitschrift, 51(1), 171–184. https://doi.org/10.1002/bimj.200810488
   PubMed Web of Science ®Google Scholar
• Austin, P. C. (2010). Statistical criteria for selecting the optimal number of untreated subjects matched to each treated subject when using many-to-one matching on the propensity score. Practice of Epidemiology, 172(9), 1092–1097. https://doi.org/10.1093/aje/kwq224
   Web of Science ®Google Scholar
• Austin, P. C. (2011). Optimal caliper widths for propensity-score matching when estimating differences in means and differences in proportions in observational studies. Pharmaceutical Statistics, 10(2), 150–161. https://doi.org/10.1002/pst.433
   PubMed Web of Science ®Google Scholar
• Austin, P. C. (2014). A comparison of 12 algorithms for matching on the propensity score. Statistics in Medicine, 33(6), 1057–1069. https://doi.org/10.1002/sim.6004
   PubMed Web of Science ®Google Scholar
• Austin, P. C., & Mamdani, M. M. (2006). A comparison of propensity score methods: A case-study estimating the effectiveness of post-AMI statin use. Statistics in Medicine, 25(12), 2084–2106. https://doi.org/10.1002/sim.2328
   Google Scholar
• Austin, P. C., & Stuart, E. A. (2015a). Moving towards best practice when using inverse probability of treatment weighting (IPTW) using the propensity score to estimate causal treatment effects in observational studies. Statistics in Medicine, 34(28), 3661–3679. https://doi.org/10.1002/sim.6607
   PubMed Web of Science ®Google Scholar
• Austin, P. C., & Stuart, E. A. (2015b). The performance of inverse probability of treatment weighting and full matching on the propensity score in the presence of model misspecification when estimating the effect of treatment on survival outcomes. Statistical Methods in Medical Research, 26(4), 1654–1670. https://doi.org/10.1177/0962280215584401
   PubMed Web of Science ®Google Scholar
• Austin, P. C., Wu, C. F., Lee, D. S., & Tu, J. V. (2020). Comparing the high-dimensional propensity score for use with administrative data with propensity scores derived from high-quality clinical data. Statistical Methods in Medical Research, 29(2), 568–588. https://doi.org/10.1177/0962280219842362
   PubMed Web of Science ®Google Scholar
• Bishop, C. D., Leite, W. L., & Snyder, P. (2018). Using propensity score weighting to reduce selection bias in large-scale data sets. Journal of Early Intervention, 40(4), 347–362. https://doi.org/10.1177/1053815118793430
   Web of Science ®Google Scholar
• Bradshaw, C. P., Pas, E. T., Musci, R. J., Kush, J. M., & Ryoo, J. H. (2021). Can policy promote adoption or outcomes of evidence-based prevention programming?: A case illustration of Positive Behavioral Interventions and Supports. Prevention Science: The Official Journal of the Society for Prevention Research, 22(7), 986–1000. https://doi.org/10.1007/s11121-021-01257-0
   PubMed Web of Science ®Google Scholar
• Cepeda, M. S., Boston, R., Farrar, J. T., & Strom, B. L. (2003). Optimal matching with a variable number of controls vs. a fixed number of controls for a cohort study: Trade-offs. Journal of Clinical Epidemiology, 56(3), 230–237. https://doi.org/10.1016/S0895-4356(02)00583-8
   PubMed Web of Science ®Google Scholar
• Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Academic Press.
   Google Scholar
• Colson, K. E., Rudolph, K. E., Zimmerman, S. C., Goin, D. G., Stuart, E. A., van der Laan, M., & Ahern, J. (2016). Optimizing matching and analysis combinations for estimating causal effects. Scientific Reports, 6, 23222. https://doi.org/10.1038/srep23222
   PubMed Web of Science ®Google Scholar
• Crump, R. K., Hotz, V. J., Imbens, G. W., & Mitnik, O. A. (2009). Dealing with limited overlap in estimation of average treatment effects. Biometrika, 96(1), 187–199. https://doi.org/10.1093/biomet/asn055
   Web of Science ®Google Scholar
• Fagan, A. A., Bumbarger, B. K., Barth, R. P., Bradshaw, C. P., Cooper, B. R., Supplee, L. H., & Walker, D. K. (2019). Scaling up evidence-based interventions in US public systems to prevent behavioral health problems: Challenges and opportunities. Prevention Science: The Official Journal of the Society for Prevention Research, 20(8), 1147–1168. https://doi.org/10.1007/s11121-019-01048-8
   PubMed Web of Science ®Google Scholar
• Fong, C., Hazlett, C., & Imai, K. (2018). Covariate balancing propensity score for continuous treatment: Application to the efficacy of political advertisements. The Annals of Applied Statistics, 12(1), 156–177. https://doi.org/10.1214/17-AOAS1101
   Web of Science ®Google Scholar
• Fuentes, A., Lüdtke, O., & Robitzsch, A. (2021). Causal inference with multilevel data: A comparison of different propensity score weighting approaches. Multivariate Behavioral Research, 1–24. https://doi.org/10.1080/00273171.2021.1925521
   Google Scholar
• Gottfredson, D. C., Cook, T. D., Gardner, F. E. M., Gorman-Smith, D., Howe, G. W., Sandler, I. N., & Zafft, K. M. (2015). Standards of evidence for efficacy, effectiveness, and scale-up research in prevention science: Next generation. Prevention Science: The Official Journal of the Society for Prevention Research, 16(7), 893–926. https://doi.org/10.1007/s11121-015-0555-x
   PubMed Web of Science ®Google Scholar
• Greifer, N. (2019). WeighIt: Weighting for covariate balance in observational studies. R package version 0.12.0. https://CRAN.R-project.org/package=WeightIt
   Google Scholar
• Greifer, N., & Stuart, E. A. (2021). Matching methods for confounders adjustment: An addition to the epidemiologist’s toolbox. Epidemiological Reviews, 43(1), 118–129. https://doi.org/10.1093/epirev/mxab003
   Google Scholar
• Gu, X. S., & Rosenbaum, P. R. (1993). Comparison of multivariate matching methods: Structures, distances, and algorithms. Journal of Computational and Graphical Statistics, 2(4), 405–420. https://doi.org/10.2307/1390693
   Google Scholar
• Hainmueller, J. (2012). Entropy balancing for causal effects: A multivariate reweighting method to produce balanced samples in observational studies. Political Analysis, 20(1), 25–46. https://doi.org/10.1093/pan/mpr025
   Web of Science ®Google Scholar
• Harder, V. S., Stuart, E. A., & Anthony, J. C. (2010). Propensity score techniques and the assessment of measured covariate balance to test causal associations in psychological research. Psychological Methods, 15(3), 234–249. https://doi.org/10.1037/a0019623
   PubMed Web of Science ®Google Scholar
• Heckman, J., & Robb, R. (1984). Alternative methods for evaluating the impact of interventions. In J. Heckman & B. Singer (Eds.), Longitudinal analysis of labor market data (pp. 156–245). Cambridge University Press.
   Google Scholar
• Hedges, L. V., & Hedberg, E. C. (2013). Intraclass correlations and covariate outcome correlations for planning two- and three-level cluster-randomized experiments in education. Evaluation Review, 37(6), 445–489. https://doi.org/10.1177/0193841X14529126
   PubMed Web of Science ®Google Scholar
• Hernán, M. A., & Robins, J. M. (2020). Causal inference: What if. Chapman & Hall/CRC.
   Google Scholar
• Ho, D., Imai, K., King, G., & Stuart, E. (2011). Matchit: Nonparametric preprocessing for parametric causal inference. Journal of Statistical Software, 42(8), 1–28. https://doi.org/10.18637/jss.v042.i08
   Web of Science ®Google Scholar
• Hong, G. (2010). Marginal mean weighting through stratification: Adjustment for selection bias in multilevel data. Journal of Educational and Behavioral Statistics, 35(5), 499–531. https://doi.org/10.3102/1076998609359785
   Web of Science ®Google Scholar
• Hong, G. (2012). Marginal mean weighting through stratification: A generalized method for evaluating multi-valued and multiple treatments with non-experimental data. Psychological Methods, 17(1), 44–60. https://doi.org/10.1037/a0024918
   PubMed Web of Science ®Google Scholar
• Keele, L. J., Lenard, M., Miratrix, L., & Page, L. (2020). Matching methods for clustered observational studies in education (EdWorkingPaper: 20-235). Annenberg Institute at Brown University. https://doi.org/10.26300/r5hw-g721
   Google Scholar
• Lee, A., & Gage, N. A. (2020). Updating and expanding systematic reviews and meta-analyses on the effects of school-wide positive behavior interventions and supports. Psychology in the Schools, 57(5), 783–804. https://doi.org/10.1002/pits.22336
   Web of Science ®Google Scholar
• Lee, B. K., Lessler, J., & Stuart, E. A. (2010). Improving propensity score weighting using machine learning. Statistics in Medicine, 29(3), 337–346. https://doi.org/10.1002/sim.3782
   PubMed Web of Science ®Google Scholar
• Lee, B. K., Lessler, J., & Stuart, E. A. (2011). Weight trimming and propensity score weighting. PLoS One, 6(3), e18174. https://doi.org/10.1371/journal.pone.0018174
   PubMed Web of Science ®Google Scholar
• Leite, W. (2017). Practical propensity score methods using R. Sage Publications.
   Google Scholar
• Leite, W. L., Aydin, B., & Gurel, S. (2019). A comparison of propensity score weighting methods for evaluating the effects of programs with multiple versions. The Journal of Experimental Education, 87(1), 75–88. https://doi.org/10.1080/00220973.2017.1409179
   Web of Science ®Google Scholar
• Lumley, T. (2004). Analysis of complex survey samples. Journal of Statistical Software, 9(8), 1–19. https://doi.org/10.18637/jss.v009.i08
   Google Scholar
• McCaffrey, D. F., Ridgeway, G., & Morral, A. R. (2004). Propensity score estimation with boosted regression for evaluating causal effects in observational studies. Psychological Methods, 9(4), 403–425. https://doi.org/10.1037/1082-989X.9.4.403
   PubMed Web of Science ®Google Scholar
• Ming, K., & Rosenbaum, P. R. (2000). Substantial gains in bias reduction from matching with a variable number of controls. Biometrics, 56(1), 118–124. https://doi.org/10.1111/j.0006-341X.2000.00118.x
   PubMed Web of Science ®Google Scholar
• Neyman, J. (1990). On the application of probability theory to agricultural experiments: Essay on principles, section 9. Statistical Science, 5, 465–480. (Original work published 1923)
   Google Scholar
• Nguyen, T., Collins, G. S., Spence, J., Daurès, J., Devereaux, P. J., Landais, P., & Manach, Y. L. (2017). Double-adjustment in propensity score matching analysis: Choosing a threshold for considering residual imbalance. BMC Medical Research Methodology, 17(1), 78. https://doi.org/10.1186/s12874-017-0338-0
   PubMedGoogle Scholar
• Office of Elementary and Secondary Education. (2021). Education innovation and research. https://oese.ed.gov/offices/office-of-discretionary-grants-support-services/innovation-early-learning/education-innovation-and-research-eir/
   Google Scholar
• Page, L. C., Lenard, M. A., & Keele, L. (2020). The design of clustered observational studies in education. AERA Open, 6(3), 233285842095414–233285842095440. https://doi.org/10.1177/2332858420954401
   Web of Science ®Google Scholar
• Pas, E. T., Ryoo, J. H., Musci, R., & Bradshaw, C. P. (2019). The effects of a state-wide scale-up of school-wide positive behavior intervention and supports on behavioral and academic outcomes: A quasi-experimental examination. Journal of School Psychology, 73, 41–55. https://doi.org/10.1016/j.jsp.2019.03.001
   PubMed Web of Science ®Google Scholar
• Pimentel, S. D., Page, L. C., Lenard, M., & Keele, L. J. (2018). Optimal multilevel matching using network flows: An application to a summer reading intervention. Annals of Applied Statistics, 12(3), 1479–1505. https://doi.org/10.1214/17-AOAS1118
   Web of Science ®Google Scholar
• R Core Team. (2021). R: A language and environment for statistical computing. https://www.R-project.org/
   Google Scholar
• Rogers, E. M. (1962). Diffusion of innovations. Free Press.
   Google Scholar
• Rosenbaum, P. R. (2020). Modern algorithms for matching in observational studies. Annual Review of Statistics and Its Application, 7(1), 143–176. https://doi.org/10.1146/annurev-statistics-031219-041058
   Google Scholar
• Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1), 41–55. https://doi.org/10.1093/biomet/70.1.41
   Web of Science ®Google Scholar
• Rubin, D. B. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66(5), 688–701. https://doi.org/10.1037/h0037350
   Web of Science ®Google Scholar
• Rubin, D. B. (1986). Comment: Which ifs have causal answers? Journal of the American Statistical Association, 81, 961–962. https://doi.org/10.1080/01621459.1986.10478355
   Web of Science ®Google Scholar
• Steiner, P. M., Cook, T. D., Shadish, W. R., & Clark, M. H. (2010). The importance of covariate selection in controlling for selection bias in observational studies. Psychological Methods, 15(3), 250–267. https://doi.org/10.1037/a0018719
   PubMed Web of Science ®Google Scholar
• Stuart, E. A. (2010). Matching methods for causal inference: A review and a look forward. Statistical Science: A Review Journal of the Institute of Mathematical Statistics, 25(1), 1–21. https://doi.org/10.1214/09-STS313
   PubMed Web of Science ®Google Scholar
• Stuart, E. A., Lee, B. K., & Leacy, F. P. (2013). Prognostic score-based balance measures can be a useful diagnostic for propensity score methods in comparative effectiveness research. Journal of Clinical Epidemiology, 66(8 Suppl), S84.e1–S90.e1. https://doi.org/10.1016/j.jclinepi.2013.01.013
   Web of Science ®Google Scholar
• Suk, Y., & Kang, H. (2021). Robust machine learning for treatment effects in multilevel observational studies under cluster-level unmeasured confounding. Psychometrika, 87(1), 310–343. https://doi.org/10.1007/s11336-021-09805-x
   Google Scholar
• Thoemmes, F. J., & Kim, E. S. (2011). A systematic review of propensity score methods in the social sciences. Multivariate Behavioral Research, 46(1), 90–118. https://doi.org/10.1080/00273171.2011.540475
   PubMed Web of Science ®Google Scholar
• Thoemmes, F., & Ong, A. D. (2016). A primer on inverse probability of treatment weighting and marginal structural models. Emerging Adulthood, 4(1), 40–59. https://doi.org/10.1177/2167696815621645
   Web of Science ®Google Scholar
• Vegetabile, B. G., Griffin, B. A., Coffman, D. L., Cefalu, M., Robbins, M. W., & McCaffrey, D. F. (2021). Nonparametric estimation of population average dose-response curves using entropy balancing weights for continuous exposures. Health Services & Outcomes Research Methodology, 21(1), 69–110. https://doi.org/10.1007/s10742-020-00236-2
   PubMed Web of Science ®Google Scholar
• Waernbaum, I. (2012). Model misspecification and robustness in causal inference: Comparing matching with doubly robust estimation. Statistics in Medicine, 31(15), 1572–1581. https://doi.org/10.1002/sim.4496
   PubMed Web of Science ®Google Scholar
• Whittaker, T. A. (2020). The comparison of latent variable propensity score models to traditional propensity score models under conditions of covariate unreliability. Multivariate Behavioral Research, 55(4), 625–646. https://doi.org/10.1080/00273171.2019.1663136
   PubMed Web of Science ®Google Scholar