Discovering Findings That Replicate From a Primary Study of High Dimension to a Follow-Up Study

Abstract

We consider the problem of identifying whether findings replicate from one study of high dimension to another, when the primary study guides the selection of hypotheses to be examined in the follow-up study as well as when there is no division of roles into the primary and the follow-up study. We show that existing meta-analysis methods are not appropriate for this problem, and suggest novel methods instead. We prove that our multiple testing procedures control for appropriate error rates. The suggested family-wise error rate controlling procedure is valid for arbitrary dependence among the test statistics within each study. A more powerful procedure is suggested for false discovery rate (FDR) control. We prove that this procedure controls the FDR if the test statistics are independent within the primary study, and independent or have positive dependence in the follow-up study. For arbitrary dependence within the primary study, and either arbitrary dependence or positive dependence in the follow-up study, simple conservative modifications of the procedure control the FDR. We demonstrate the usefulness of these procedures via simulations and real data examples. Supplementary materials for this article are available online.

Keywords:

• False discovery rate
• Genome-wide association studies
• Meta-analysis
• Multiple comparisons
• Replicability analysis

Acknowledgments

This work was supported by grant no. 2012896 from the Israel Science Foundation (ISF). The authors thank Yoav Benjamini, Daniel Yekutieli, and the referees for helpful comments.