Bad numbers: coping with flawed decision support

Abstract

Evidence is accumulating that many spreadsheet-based decision support systems contain errors. These errors can result in bad numbers, which in turn could lead to bad decisions. We review the literature on the origins and consequences of bad (erroneous) numbers produced by models and/or decision support systems built around them. Then, we present a case study in which an experiment in visual support for a sequential decision-making task was tainted by bad numbers. Both the literature review and the experiment indicate a robust human ability to overcome flawed decision support. We conclude with questions that need to be addressed in order to better understand the capabilities of humans to deal with erroneous results from decision support systems.

Keywords:

• decision support systems
• practice of OR
• human–machine systems

Acknowledgements

This work was supported by US National Science Foundation Grants SBS-9730465 and SES-9730465. We thank the following for help with conducting and analyzing the experiments: Chandra Benjamin, Linnda Caporael, Nick Clesceri, Francisco Ferrer, Ted Schuster, and Tom Zimmie. The following helped with computational aspects of the work: Mihoko Bennett, Kevin Brewer, Mark Miller, and Carlos Ortiz. We benefited from discussions with Peter Diggle, Mike Pidd, Nong Shang, and Langdon Winner. We also thank the anonymous subjects who played the WhereNext game. TR Willemain acknowledges support provided by the Department of Management Science, Lancaster University, England. The work of KR Fleischmann was supported under a National Science Foundation Graduate Research Fellowship.