COMPARING EMPIRICAL DISTRIBUTIONS OF P-VALUES FROM SIMULATIONS

Abstract

In the context of hypothesis testing simulation studies, this paper advocates using graphical and numerical tools to summarize the results, beyond the conventional practice of just reporting empirical levels. Some of these tools are defined by us. They are mainly based on the computation of distances between the empirical distribution function of the p-values derived from the simulation experiment and the distribution function of the U([0, 1]) random variable. Their null distribution is tabulated. A joint study of several distances reveals that important aspects of a test can pass unnoticed if only empirical significance levels are calculated. The proposed tools are applied to two practical examples, demonstrating their usefulness in the discrimination between alternative test procedures and also in the detection of data not according to the null hypothesis.

Keywords:

• Distances between distribution functions
• Goodness of fit
• Graphics
• Tests of equality of variances
• Tests of uncorrelation

ACKNOWLEDGMENTS

The authors are grateful to an Associate Editor for his/her constructive comments and suggestions. The main part of the work was done when the first author was at the Universitat Pompeu Fabra. This work was partially supported by the Spanish DGES grant PB96-0300.