Making causal inferences about treatment effect sizes from observational datasets

Abstract

In the era of big data and cloud computing, analysts need statistical models to go beyond predicting outcomes to forecasting how outcomes change when decision-makers intervene to change one or more causal factors. This paper reviews methods to estimate the causal effects of treatment choices on patient health outcomes using observational datasets. Methods are limited to those that model choice of treatment (propensity scoring) and treatment outcomes (instrumental variable, difference in differences, control function). A regression framework was developed to show how unobserved confounding covariates and heterogeneous outcomes can introduce biases to effect size estimates. In response to criticisms that outcome approaches are not systematic and subject to model misspecification error, we extend the control function approach of Lu and White by applying Best Approximating Model technology (BAM-CF). Results from simulation experiments are presented to compare biases between BAM-CF and propensity scoring in the presence of an unobserved confounder. We conclude no one strategy is ‘optimal’ for all datasets, and analyst should consider multiple approaches to assess robustness. For both observational and randomized datasets, researchers should assess how moderating covariates impact estimates of treatment effect sizes so that clinicians can understand what is best for each individual patient.

KEYWORDS:

• Causal modeling
• control functions
• propensity scoring
• endogeneity
• moderator analyses

Acknowledgements

The authors wish to express their appreciation to Tracy McKay, M.S.H.I., B.S., Elena V. Perez, B.S.H.A., with the Center for Advanced Statistics in Education at the Loma Linda VA Medical Center, Loma Linda, California, and Annie B. Wicker, B.S., with the Data Management and Support Center with the Office of Academic Affiliations, Department of Veterans Affairs, St. Louis, MO. Sincere gratitude is expressed for the support from the Office of Academic Affiliations (OAA) Data Management and Support Center (Christopher T. Clarke, Ph.D., David S. Bernett, and Laura Stefanowycz) in St. Louis MO.

Disclosure statement

No potential conflict of interest was reported by the authors.

Additional information

Funding

This study was supported in part by Health Services Research, Department of Veterans Affairs (SDR#13-426, IIR#14-071, and IIR#15-084) and the Office of Academic Affiliations, Veterans Health Administration, Department of Veterans Affairs, Washington, DC. All statements and descriptions expressed herein do not necessarily reflect the opinions or positions of the Department of Veterans Affairs or its affiliated institutions.

Notes on contributors

T. Michael Kashner

T. Michael Kashner, Ph.D., J.D., is a Health Econometrician with degrees in Law and Public Health, and Senior Scientist with the Veterans Health Administration's Office of Academic Affiliations (OAA) in the Department of Veterans Affairs (VA), Washington, DC, and Research Professor of Medicine at the Loma Linda University School of Medicine in Loma Linda, CA. As a well-funded and published health services researcher since 1983, Dr Kashner applies advanced analytic methods to large education, clinical, cost, and administrative databases to make robust causal inferences in the areas of health professions education. As former Research Professor of Psychiatry at the University of Texas Southwestern Medical School, he was involved in numerous studies assessing the effectiveness of mental health services where he was also a former VA Research Career Scientist with the Department of Veterans.

Steven S. Henley

Steven S. Henley, M.S., is a Computational Statistician, Research Professor of Medicine at the Loma Linda University School of Medicine, and President of Martingale Research Corporation. Professor Henley has extensive experience in statistical modeling, algorithm development, machine learning, systems engineering, and R&D project management. His research focuses on the development and evaluation of new theory, algorithms, and statistical software for clinical, epidemiologic, and health services research. Prof. Henley is a principal investigator for National Institutes of Health (NIH) Small Business Innovation Research (SBIR) sponsored research that develops advanced statistical modeling technologies.

Richard M. Golden

Richard M. Golden, Ph.D., M.S.E.E., B.S.E.E., is a Mathematical Psychometrician, Professor of Cognitive Science and Electrical Engineering, and Program Head of the Undergraduate Cognitive Science and Graduate Applied Cognition and Neuroscience Programs in the School of Behavioral and Brain Sciences at the University of Texas at Dallas (UTD). He was an action editor of the Journal of Mathematical Psychology, author of Mathematical Methods for Neural Network Analysis and Design (MIT Press, 1996), and has published numerous papers on assessing model quality and robust statistical inference in the possible presence of model misspecification. Dr Golden has been a principal investigator on National Science Foundation (NSF) funded research and collaborated on National Institutes of Health (NIH) Small Business Innovation Research (SBIR) projects that develop novel statistical modeling methods.

Xiao-Hua Zhou

Xiao-Hua (Andrew) Zhou, Ph.D., is Boya Chair Professor in Beijing International Center for Mathematical Research and Chair of the Department of Biostatistics at Peking University. He is also a former Professor of the Department of Biostatistics at University of Washington and former Director of the Unit of Biostatistics at U.S. Department of Veterans Affairs Seattle Center of Innovation for Veteran-Centered and Value-Driven Care. He was awarded a Research Career Scientist Award by the U.S. Department of Veterans Affairs. He has made important contributions to medicine and public health by developing new statistical methods, particularly in diagnostic medicine and causal inference. Specifically, he has developed new statistical methods for (1) prediction and modeling of cost data, (2) causal inferences in broken clinical trials, such non-compliance and truncation by death, (3) studies on the accuracy of diagnostic tests, and (4) statistical methods for precision medicine. These statistical problems originate from collaborative research that he had been doing with his medical investigators. He has published over 245 statistical methodology and medical papers and is either the corresponding author or senior author on most of them; many of them have been published in top statistical journals, such as Journal of the Royal Statistical Society Series B (JRSS B), Journal of the American Statistical Association (JASA), Biometrics, Biometrika, Annals of Statistics, and Statistics in Medicine.