Quickly Assessing Contributions to Input Uncertainty

Abstract

“Input uncertainty” refers to the (often unmeasured) variability in simulation-based performance estimators that is a consequence of driving the simulation with input models (e.g., fully specified univariate distributions of i.i.d. inputs) that are based on real-world data. In 2012 Ankenman and Nelson presented a quick-and-easy diagnostic experiment to assess the overall effect of input uncertainty on simulation output. When their method reveals that input uncertainty is substantial, then the natural next questions are which input distributions contribute the most to input uncertainty, and from which input distributions would it be most beneficial to collect more data? They proposed a possibly lengthy sequence of additional diagnostic experiments to answer these questions. In this paper we provide a method that obtains an estimator of the overall variance due to input uncertainty, the relative contribution to this variance of each input distribution, and a measure of the sensitivity of overall uncertainty to increasing the real-world sample-size used to fit each distribution, all from a single diagnostic experiment. Our approach exploits a metamodel that relates the means and variances of the input distributions to the mean response of the simulation output, and bootstrapping of the real-world data to represent input-model uncertainty. Further, we investigate whether and how the simulation outputs from the nominal and diagnostic experiments may be combined to obtain a better performance estimator. For the case when the analyst obtains additional real-world data, refines the input models, and runs a follow-up experiment, we analyze whether and how the simulation outputs from all three experiments should be combined. Numerical illustrations are provided.

Keywords::

• Stochastic simulation
• input modeling
• input uncertainty
• output analysis

Additional information

Notes on contributors

Eunhye Song

EUNHYE SONGis a Ph.D. candidate of the Department of Industrial Engineering and Management Sciences at Northwestern University. Her research interests are input uncertainty quantification, design of simulation experiments and simulation optimization. Her current research is focused on simulation optimization under input uncertainty.

Barry L. Nelson

BARRY L. NELSONis the Walter P. Murphy Professor of the Department of Industrial Engineering and Management Sciences at Northwestern University. He is a Fellow of INFORMS and IIE. His research centers on the design and analysis of computer simulation experiments on models of stochastic systems, and he is the author of Foundations and Methods of Stochastic Simulation: A First Course, from Springer.