The tradeoff between sampling and switching: New sequential procedures for indifference-zone selection

Statistical Ranking and Selection (R&S) is a collection of experiment design and analysis techniques for selecting the “population” with the largest or smallest mean performance from among a finite set of alternatives. R&S procedures have received considerable research attention in the stochastic simulation community, and they have been incorporated in commercial simulation software. One of the ways that R&S procedures are evaluated and compared is via the expected number of samples (often replications) that must be generated to reach a decision. In this paper we argue that sampling cost alone does not adequately characterize the efficiency of ranking-and-selection procedures, and the cost of switching among the simulations of the alternative systems should also be considered. We introduce two new, adaptive procedures, the minimum switching sequential procedure and the multi-stage sequential procedure with tradeoff, that provide the same statistical guarantees as existing procedures and significantly reduce the expected total computational cost of application, especially when applied to favorable configurations of the competing means.

Notes