Accurate and Ultra-Efficient p-Value Calculation for Higher Criticism Tests

Abstract

In modern data science, higher criticism (HC) method is effective for detecting rare and weak signals. The computation, however, has long been an issue when the number of p-values combined (K) and/or the number of repeated HC tests (N) are large. Some computing methods have been developed, but they all have significant shortcomings, especially when a stringent significance level is required. In this article, we propose an accurate and highly efficient computing strategy for four variations of HC. Specifically, we propose an unbiased cross-entropy-based importance sampling method (${\text{IS}}_{CE}$) to benchmark all existing computing methods, and develop a modified SetTest method (MST) that resolves numerical issues of the existing SetTest approach. We further develop an ultra-fast approach (UFI) combining pre-calculated statistical tables and cubic spline interpolation. Finally, following extensive simulations, we provide a computing strategy integrating MST, UFI, and other existing methods with R package “HCp” for virtually any K and small p-values ($\sim {10}^{-20}$). The method is applied to a COVID-19 disease surveillance example for spatio-temporal outbreak detection from case numbers of 804 days in 3342 counties in the United States. Results confirm viability of the computing strategy for large-scale inferences. Supplementary materials for this article are available online.

KEYWORDS:

• Analytical approximation
• Asymptotic rare and weak model
• Higher criticism
• Importance sampling
• p-value computation

Supplementary Materials

Additional results and proofs:Include the analytic derivation of formulas and extensive simulation results. (Appendix.pdf, PDF file)

Data and code:Contain the R package (also available on GitHub), the R code and data (in “Simulation” sub-folder) to reproduce the results presented in the article, and README file (README.md, markdown file) to give detailed instructions on how to install the package and descriptions of each function. (HCp.zip, zipped file)

Disclosure Statement

The authors report there are no competing interests to declare.

Additional information

Funding

WW, YF and GCT are partially funded by NIH R01LM014142.