Robustness of a multiple ranking procedure: a monte carlo experiment illustrating design and analysis techniques

Abstract

This case study demonstrates statistical design and analysis techniques applicable to any Monte Carlo or simulation experiment, namely a 2⁷⁻³ experimental design, antithetic variates, sample size determination, analysis of variance, regression analysis, and simultaneous inference. The example is a Monte Carlo investigation of the robustness of Bechhofer and Blumenthal’s multiple ranking procedure (MRP). The investigation shows that their procedure works often, but not always. Factors that make it break down, are identified.

Keywords:

• simulation
• regression analysis
• analysis of variance
• antithetic variables
• sample size
• simultaneous inference