Confidence intervals with maximal average power

Abstract

We propose a frequentist testing procedure that maintains a defined coverage and is optimal in the sense that it gives maximal power to detect deviations from a null hypothesis when the alternative to the null hypothesis is sampled from a pre-specified distribution (the prior distribution). Selecting a prior distribution allows to tune the decision rule. This leads to an increased power, if the true data generating distribution happens to be compatible with the prior. It comes at the cost of losing power, if the data generating distribution or the observed data are incompatible with the prior. We illustrate the proposed approach for a binomial experiment, which is sufficiently simple such that the decision sets can be illustrated in figures, which should facilitate an intuitive understanding. The potential beyond the simple example will be discussed: the approach is generic in that the test is defined based on the likelihood function and the prior only. It is comparatively simple to implement and efficient to execute, since it does not rely on Minimax optimization. Conceptually it is interesting to note that for constructing the testing procedure the Bayesian posterior probability distribution is used.

Keywords:

• Confidence interval
• binomial distribution
• exact test
• Bayesian posterior
• optimal average power

Acknowledgments

Johanna Mielke was supported by the Swiss State Secretariat for Education, Research and Innovation (SERI) under contract number 999754557. The opinions expressed and arguments employed herein do not necessarily reflect the official views of the Swiss Government. This project is part of the IDEAS European training network (http://www.ideas-itn.eu/) from the European Union’s Horizon 2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No 633567. We would like to thank Prof. Byron Jones for encouraging this work.