Multi-model Markov decision processes

Abstract

Markov decision processes (MDPs) have found success in many application areas that involve sequential decision making under uncertainty, including the evaluation and design of treatment and screening protocols for medical decision making. However, the data used to parameterize the model can influence what policies are recommended, and multiple competing data sources are common in many application areas, including medicine. In this article, we introduce the Multi-model Markov decision process (MMDP) which generalizes a standard MDP by allowing for multiple models of the rewards and transition probabilities. Solution of the MMDP generates a single policy that maximizes the weighted performance over all models. This approach allows the decision maker to explicitly trade-off conflicting sources of data while generating a policy of the same level of complexity for models that only consider a single source of data. We study the structural properties of this problem and show that it is at least NP-hard. We develop exact methods and fast approximation methods supported by error bounds. Finally, we illustrate the effectiveness and the scalability of our approach using a case study in preventative blood pressure and cholesterol management that accounts for conflicting published cardiovascular risk models.

Keywords:

• Dynamic programming
• medical decision making
• Markov decision processes
• parameter ambiguity
• healthcare applications

Additional information

Funding

This work was supported by the National Science Foundation under grant numbers DGE-1256260 (Steimle) and CMMI-1462060 (Denton); any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.

Notes on contributors

Lauren N. Steimle

Lauren N. Steimle is an assistant professor in the H. Milton Stewart School of Industrial and Systems Engineering at Georgia Institute of Technology. She received her PhD and MSE in industrial and operations engineering from the University of Michigan and her BS in systems science and engineering from Washington University in St. Louis. Her research interests include data-driven optimization and stochastic modeling with applications to medicine and public health.

David L. Kaufman

David Kaufman is an assistant professor at the University of Michigan-Dearborn, College of Business, where he teaches courses in Decision Sciences. He holds a PhD in Industrial and Operations Engineering from the University of Michigan. His research interests are in stochastic processes and decision models for systems where variability and uncertainty play an important role in design, analysis, and management.

Brian T. Denton

Brian Denton is Chair of the Department of Industrial and Operations Engineering at the University of Michigan. His research interests are in data-driven sequential decision making and optimization under uncertainty with applications to medicine. Before joining the University of Michigan he worked at IBM, Mayo Clinic, and North Carolina State University. He is an INFORMS Fellow, past Chair of the INFORMS Health Applications Section, and he is Past President of INFORMS.