Publication Policies for Replicable Research and the Community-Wide False Discovery Rate

Abstract

Recent literature has shown that statistically significant results are often not replicated because the “p-value < 0.05” publication rule results in a high false positive rate (FPR) or false discovery rate (FDR) in some scientific communities. While recommendations to address the phenomenon vary, many amount to incorporating additional study summary information, such as prior null hypothesis odds and/or effect sizes, in some way. This article demonstrates that a statistic called the local false discovery rate (lfdr), which incorporates this information, is a sufficient summary for addressing false positive rates. Specifically, it is shown that lfdr-values among published results are sufficient for estimating the community-wide FDR for any well-defined publication policy, and that lfdr-values are sufficient for defining policies for community-wide FDR control. It is also demonstrated that, though p-values can be useful for computing an lfdr, they alone are not sufficient for addressing the community-wide FDR. Data from the recent replication study are used to compare publication policies and illustrate the FDR estimator.

Keywords:

• False discovery rate
• False positive
• p-Value
• Local false discovery rate
• Replicable research
• Statistical significance

Supplementary Materials

The supplemental file contains R code for data analysis in Section 6.

Acknowledgments

The authors thank to anonymous referees, AEs and the editor for helpful comments that lead to a much improved article.

Notes

1 Self-contained decision rules are referred to as simple decision rules in decision theory and multiple testing (Robbins 1951; Sun and Cai 2007). We do not adopt this terminology here because decisions within each study need not be simple, as in the Empirical Bayes example in Section 2.