Accumulation Tests for FDR Control in Ordered Hypothesis Testing

ABSTRACT

Multiple testing problems arising in modern scientific applications can involve simultaneously testing thousands or even millions of hypotheses, with relatively few true signals. In this article, we consider the multiple testing problem where prior information is available (for instance, from an earlier study under different experimental conditions), that can allow us to test the hypotheses as a ranked list to increase the number of discoveries. Given an ordered list of n hypotheses, the aim is to select a data-dependent cutoff k and declare the first k hypotheses to be statistically significant while bounding the false discovery rate (FDR). Generalizing several existing methods, we develop a family of “accumulation tests” to choose a cutoff k that adapts to the amount of signal at the top of the ranked list. We introduce a new method in this family, the HingeExp method, which offers higher power to detect true signals compared to existing techniques. Our theoretical results prove that these methods control a modified FDR on finite samples, and characterize the power of the methods in the family. We apply the tests to simulated data, including a high-dimensional model selection problem for linear regression. We also compare accumulation tests to existing methods for multiple testing on a real data problem of identifying differential gene expression over a dosage gradient. Supplementary materials for this article are available online.

KEYWORDS:

• Accumulation test
• False discovery rate
• Multiple testing
• Ordered hypothesis testing
• Sequential hypothesis testing

Supplementary Materials

In the supplementary materials we provide additional plots for further examination of the results of our simulated data experiments (Section A) and proofs for all the theoretical results presented in the paper (Section B).

Notes

1 http://www.stat.uchicago.edu/∼rina/accumulationtests.html

2 There is one minor caveat here: if the p-values are conservative then it’s possible to have $\mathbb{P}\left\{p_i=1\right\}>0$; in this case, accumulation functions such as ForwardStop or HingeExp will fail, since h(1) = +∞. In practice, for any such choice of h, we would want to truncate h at some large but finite value, that is, replace h(t) with h(t)∧C for some large positive C. Since this will reduce the expected value only slightly, that is ∫¹ _{t = 0}h(t)∧C dt is only slightly smaller than 1, the FDR and FDP bounds would only be slightly worse.

3 Note that the second assumption in (17(17) $$\begin{array}{c}\hfill \left\{\begin{array}{c}\hfill f\text{is}\text{monotone}\text{nonincreasing}\text{over}t\in [0,1],\\ \hfill f\left(t\right)\le f(t-{\Delta }_t)-\nu {\Delta }_t\text{for}\text{all}t\text{such}\text{that}f\left(t\right)\\ \hfill \ge 1-\alpha \text{and}0\le {\Delta }_t\le \delta ,\\ \hfill t\mapsto t·f\left(t\right)\text{is}\text{a}\text{nondecreasing}\text{function.}\end{array}\right.\end{array}$$(17) ) ensures that T is uniquely defined in each of these cases, that is in the middle case where $f\left(1\right)<\frac{{\mu }_0-\alpha }{{\mu }_0-{\mu }_1}<f\left(0\right)$, the inverse f ^{− 1} is well-defined at the value where it is applied.

4 http://www.stat.uchicago.edu/∼rina/accumulationtests.html

5 Some additional plots exploring these results are in the supplementary materials.

6 Some additional plots exploring these results are in the supplementary materials.

7 http://www.stat.uchicago.edu/∼rina/accumulationtests.html

8 While several existing methods for multiple testing yield guarantees for FDR control even in the case of dependent p-values, the methods we are aware of are quite conservative and yield nearly zero power in this experiment. Specifically, Benjamini and Yekutieli’s modification (2001) of the BH method, the Holm-Bonferroni method (Holm 1979), and the Bonferroni correction each yielded, even at target FDR level α = 0.9, no more than two discoveries for this gene expression experiment; in contrast, the accumulation test methods examined here yield thousands of discoveries.

9 Data available at http://www.ncbi.nlm.nih.gov/sites/GDSbrowser?acc=GDS2324 or via the GEOquery package (Davis and Meltzer 2007) in R.