Data-driven stochastic optimization approaches to determine decision thresholds for risk estimation models

Abstract

The increasing availability of data has popularized risk estimation models in many industries, especially healthcare. However, properly utilizing these models for accurate diagnosis decisions remains challenging. Our research aims to determine when a risk estimation model provides sufficient evidence to make a positive or negative diagnosis, or if the model is inconclusive. We formulate the Two-Threshold Problem (TTP) as a stochastic program which maximizes sensitivity and specificity while constraining false-positive and false-negative rates. We characterize the optimal solutions to TTP as either two-threshold or one-threshold and show that its optimal solution can be derived from a related linear program (TTP*). We also derive utility-based and multi-class classification frameworks for which our analytical results apply. We solve TTP* using data-driven methods: quantile estimation (TTP*-Q) and distributionally robust optimization (TTP*-DR). Through simulation, we characterize the feasibility, optimality, and computational burden of TTP*-Q and TTP*-DR and compare TTP*-Q to an optimized single threshold. Finally, we apply TTP* to concussion assessment data and find that it achieves greater accuracy at lower misclassification rates compared with traditional approaches. This data-driven framework can provide valuable decision support to clinicians by identifying “easy” cases which can be diagnosed immediately and “hard” cases which may require further evaluation before diagnosing.

Keywords:

• Data-driven optimization
• stochastic programming
• distributionally robust optimization
• quantile estimation
• diagnosis decisions
• risk estimation models

View correction statement:

Correction

Acknowledgments

The authors thank two anonymous referees and an associate editor, whose comments have improved the paper.

Additional information

Funding

This material is based upon work supported by the National Science Foundation Graduate Research Fellowship under Grant No. DGE 1256260. This work has also been supported by National Science Foundation under Grant no. CMMI-1662774. This publication was made possible, in part, with support from the Grand Alliance Concussion Assessment, Research, and Education (CARE) Consortium, funded, in part, by the National Collegiate Athletic Association (NCAA) and the Department of Defense (DOD). The U.S. Army Medical Research Acquisition Activity, 820 Chandler Street, Fort Detrick MD 21702-5014 is the awarding and administering acquisition office. This work was supported by the Office of the Assistant Secretary of Defense for Health Affairs through the Psychological Health and Traumatic Brain Injury Program under Award No. W81XWH-14-2-0151. Opinions, interpretations, conclusions and recommendations are those of the author(s) and are not necessarily endorsed by the Department of Defense (DHP funds).

Notes on contributors

Gian-Gabriel P. Garcia

Gian-Gabriel Garcia is a Ph.D. candidate in the Industrial and Operations Engineering Department at the University of Michigan. He holds a Bachelor’s degree in industrial engineering from the University of Pittsburgh and a Master’s degree in industrial and operations Engineering from the University of Michigan. In his research, Gian is interested in developing data-driven frameworks using predictive and prescriptive analytics to address by high-impact problems in healthcare. Motivated by applications in concussion, glaucoma, and cardiovascular disease, his current research focuses on (i) using large clinical datasets to gain patient-specific insights on disease progression and (ii) combining these insights with stakeholders’ perspectives to improve diagnosis and treatment decisions. This research has been recognized by the National Science Foundation Graduate Research Fellowship, the INFORMS Bonder Scholarship for Applied Operations Research in Health Services, the SMDM Lee B. Lusted Prize in Quantitative Methods and Theoretical Developments, and first prize at the INFORMS Minority Issues Forum Poster Competition. Gian has also been awarded the Richard F. and Eleanor A. Towner Prize for Distinguished Academic Achievement from the College of Engineering at the University of Michigan.

Mariel S. Lavieri

Mariel S. Lavieri is an associate professor in the Industrial and Operations Engineering Department at the University of Michigan. She has bachelor’s degrees in industrial and systems engineering and statistics and a minor in string bass performance from the University of Florida. She holds a master’s and Ph.D. in management science from the University of British Columbia. In her work, she applies operations research to healthcare topics. Among others, she has developed dynamic programming, stochastic control, and continuous, partially observable state space models to guide screening, monitoring, and treatment decisions of chronic disease patients. She has also created models for health workforce and capacity planning. Among others, she is the recipient of the Willie Hobbs Moore Aspire, Advance, Achieve Mentoring Award, the National Science Foundation CAREER Award, the International Conference on Operations Research Young Participant with Most Practical Impact Award, and the Bonder Scholarship. She has also received the Pierskalla Best Paper Award, and an honorary mention in the George B. Dantzig Dissertation Award. She has guided work that won the Medical Decision Making Lee Lusted Award, the INFORMS Doing Good with Good OR Award, the IBM Research Service Science Best Student Paper Award and the Production and Operations Management Society College of Healthcare Operations Management Best Paper Award.

Ruiwei Jiang

Ruiwei Jiang received the B.S. degree in industrial engineering from Tsinghua University, Beijing, China, in 2009, and a Ph.D. degree in industrial and systems engineering from the University of Florida, Gainesville, FL, in 2013. He is currently an assistant professor with the Department of Industrial and Operations Engineering, University of Michigan, Ann Arbor, MI. His research interests include stochastic optimization, integer programming, and their applications in power system and healthcare operations.

Michael A. McCrea

Michael A. McCrea is tenured professor, eminent scholar, and vice chair of research in the Department of Neurosurgery at the Medical College of Wisconsin (MCW), where he also serves as Co-Director for the MCW Center for Neurotrauma Research. He has an appointment as a research neuropsychologist at the Clement Zablocki VA Medical Center in Milwaukee, Wisconsin. Dr. McCrea has been an active researcher in the neurosciences, with numerous scientific publications, book chapters, and national and international lectures on the topic of traumatic brain injury. He authored the text Mild Traumatic Brain Injury and Postconcussion Syndrome: The New Evidence Base for Diagnosis and Treatment published by Oxford University Press. Dr. McCrea has led several large, multi-center studies on the effects of traumatic brain injury and sport-related concussion. He currently is co-PI on the NCAA-DoD CARE Consortium and several other large-scale studies investigating the acute and chronic effects of TBI in various populations at risk. Dr. McCrea is also a key investigator on the TRACK-TBI and TBI Endpoint Development studies of civilian brain injury.

Thomas W. McAllister

Thomas W. McAllister, MD, is the Albert Eugene Sterne Professor and Chairman, Indiana University School of Medicine Department of Psychiatry. He was previously Millennium Professor of Psychiatry and Neurology, Director of the Section of Neuropsychiatry at Dartmouth Medical School and Vice Chair for Neuroscience Research for the Department of Psychiatry. He is a past president of the American Neuropsychiatric Association. Dr. McAllister received his undergraduate degree from Dartmouth College and his medical degree from Dartmouth Medical School. He served on the faculties of the University of Kentucky and the University of Pennsylvania before returning to Dartmouth Medical School in 1990. Dr. McAllister has been working in the field of brain injury recovery for over 25 years. He has written widely on the neuropsychiatric sequelae of TBI and has been the principal investigator of numerous grants from NIH, the CDC, NOCSAE, and the Department of Defense (DoD), exploring the nature of cognitive and behavioral difficulties following mild and moderate TBI. With Drs. Jon Silver and Stuart Yudofsky he is a co-editor of the Textbook of Traumatic Brain Injury published by American Psychiatric Publishing, Inc. Recent research has focused on characterizing the biomechanical basis of concussion, and the effects of repetitive head impacts on brain structure and function in contact sport athletes.

Steven P. Broglio

Steven Broglio is a professor of kinesiology, neurology, and physical medicine and rehabilitation at the University of Michigan in Ann Arbor. Dr. Broglio completed his training at the University of Georgia, followed by his first faculty position at the University of Illinois at Urbana-Champaign. He has been at the University of Michigan since 2011. At Michigan, Dr. Broglio is the Director of the Michigan Concussion Center and the NeuroTrauma Research Laboratory where he oversees clinical care, educational outreach, and multi-disciplinary research aimed at fundamental questions on concussion prevention, identification, diagnosis, management, and outcomes. His research has been supported by numerous foundations and federal funding agencies, resulting in nearly 150 peer reviewed works. Dr. Broglio is a co-PI on the CARE Consortium, the largest prospective investigation of concussion ever conducted.