General-purpose ranking and selection for computer simulation

ABSTRACT

Many indifference-zone Ranking-and-Selection (R&S) procedures have been invented for choosing the best simulated system. To obtain the desired Probability of Correct Selection (PCS), existing procedures exploit knowledge about the particular combination of system performance measure (e.g., mean, probability, variance, quantile) and assumed output distribution (e.g., normal, exponential, Poisson). In this article, we take a step toward general-purpose R&S procedures that work for many types of performance measures and output distributions, including situations where different simulated alternatives have entirely different output distribution families. There are only two versions of our procedure: with and without the use of common random numbers. To obtain the required PCS we exploit intense computation via bootstrapping, and to mitigate the computational effort we create an adaptive sample-allocation scheme that guides the procedure to quickly reach the necessary sample size. We establish the asymptotic PCS of these procedures under very mild conditions and provide a finite-sample empirical evaluation of them as well.

KEYWORDS:

• Simulation
• optimization
• ranking and selection

Acknowledgements

The authors thank the Associate Editor and two referees for their feedback.

Funding

This research was supported by the Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Science, ICT&Future Planning 2014R1A1A3049955, and by Hankuk University of Foreign Studies Research Fund. Portions of this work were published in the Proceedings of the 2014 Winter Simulation Conference as Lee and Nelson (2014).

Additional information

Notes on contributors

Soonhui Lee

Soonhui Lee is an Assistant Professor in the College of Business at Hankuk University of Foreign Studies. She received her B.S. at KAIST, M.S. at Georgia Institute of Technology, and Ph.D. in Industrial Engineering and Management Sciences at Northwestern University. Her research interests include stochastic optimization and its application.

Barry L. Nelson

Barry L. Nelson is the Walter P. Murphy Professor in the Department of Industrial Engineering and Management Sciences at Northwestern University. He is a Fellow of INFORMS and IIE. His research centers on the design and analysis of computer simulation experiments on models of stochastic systems, and he is the author of Foundations and Methods of Stochastic Simulation: A First Course, published by Springer.