Multiple Testing When Many p-Values are Uniformly Conservative, with Application to Testing Qualitative Interaction in Educational Interventions

References

• Arias-Castro, E., Candès, E. J., and Plan, Y. (2011), “Global Testing Under Sparse Alternatives: ANOVA, Multiple Comparisons and the Higher Criticism,” Annals of Statistics, 39, 2533–2556.
   Web of Science ®Google Scholar
• Bangert-Drowns, R. L., Hurley, M. M., and Wilkinson, B. (2004), “The Effects of School-Based Writing-to-Learn Interventions on Academic Achievement: A Meta-Analysis,” Review of Educational Research, 74, 29–58.
   Web of Science ®Google Scholar
• Benjamini, Y. (2010), “Simultaneous and Selective Inference: Current Successes and Future Challenges,” Biometrical Journal, 52, 708–721.
   PubMed Web of Science ®Google Scholar
• Benjamini, Y., and Hochberg, Y. (1995), “Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing,” Journal of the Royal Statistical Society, Series B, 57, 289–300.
   Google Scholar
• Berger, R. L. (1982), “Multiparameter Hypothesis Testing and Acceptance Sampling,” Technometrics, 24, 295–300.
   Web of Science ®Google Scholar
• Bloom, H. S., Raudenbush, S. W., Weiss, M. J., and Porter, K. (2017), “Using Multisite Experiments to Study Cross-Site Variation in Treatment Effects: A Hybrid Approach with Fixed Intercepts and a Random Treatment Coefficient,” Journal of Research on Educational Effectiveness, 10, 817–842.
   Google Scholar
• Chernozhukov, V., Chetverikov, D., and Kato, K. (2013), “Testing Many Moment Inequalities,” arXiv: 1312.7186.
   Google Scholar
• Cooper, H., Valentine, J. C., Charlton, K., and Melson, A. (2003), “The Effects of Modified School Calendars on Student Achievement and on School and Community Attitudes,” Review of Educational Research, 73, 1–52.
   Web of Science ®Google Scholar
• CRASH-2-Collaborators (2011), “The Importance of Early Treatment with Tranexamic Acid in Bleeding Trauma Patients: An Exploratory Analysis of the CRASH-2 randomised Controlled Trial,” The Lancet, 377, 1096–1101.
   PubMed Web of Science ®Google Scholar
• Cronbach, L. J., and Snow, R. E. (1977), Aptitudes and Instructional Methods: A Handbook for Research on Interactions, New York: Irvington.
   Google Scholar
• Donoho, D., and Jin, J. (2004), “Higher Criticism for Detecting Sparse Heterogeneous Mixtures,” Annals of Statistics, 32, 962–994.
   Web of Science ®Google Scholar
• Fan, J., Hall, P., and Yao, Q. (2007), “To How Many Simultaneous Hypothesis Tests can Normal, Student’s t or Bootstrap Calibration be Applied?” Journal of the American Statistical Association, 102, 1282–1288.
   Google Scholar
• Fisher, R. A. (1925), Statistical Methods for Research Workers, Edinburgh, UK: Oliver and Boyd.
   Google Scholar
• Fithian, W., Sun, D., and Taylor, J., (2014), “Optimal Inference After Model Selection,” arXiv:1410.2597.
   Google Scholar
• Gail, M., and Simon, R. (1985), “Testing for Qualitative Interactions Between Treatment Effects and Patient Subsets,” Biometrics, 41, 361–372.
   PubMed Web of Science ®Google Scholar
• Hansen, P. R. (2005), “A Test for Superior Predictive Ability,” Journal of Business and Economic Statistics, 23, 365–380.
   Web of Science ®Google Scholar
• Heller, R., Rosenbaum, P. R., and Small, D. S. (2009), “Split Samples and Design Sensitivity in Observational Studies,” Journal of the American Statistical Association, 104, 1090–1101.
   Web of Science ®Google Scholar
• Holm, S. (1979), “A Simple Sequentially Rejective Multiple Test Procedure,” Scandinavian Journal of Statistics, 6, 65–70.
   Web of Science ®Google Scholar
• Konstantopoulos, S. (2011), “Fixed Effects and Variance Components Estimation in Three-Level Meta-Analysis,” Research Synthesis Methods, 2, 61–76.
   PubMed Web of Science ®Google Scholar
• Lehmann, E. (1952), “Testing Multiparameter Hypotheses,” Annals of Mathematical Statistics, 23, 541–552.
   Google Scholar
• Lei, L., Ramdas, A., and Fithian, W. (2017), “STAR: A General Interactive Framework for FDR Control Under Structural Constraints,” arXiv:1710.02776.
   Google Scholar
• Pan, G., and Wolfe, D. A. (1997), “Test for Qualitative Interaction of Clinical Significance,” Statistics in Medicine, 16, 1645–1652.
   PubMed Web of Science ®Google Scholar
• Pizzocaro, G., Piva, L., Colavita, M., Ferri, S., Artusi, R., Boracchi, P., Parmiani, G., and Marubini, E. (2001), “Interferon Adjuvant to Radical Nephrectomy in Robson Stages II and III Renal Cell Carcinoma: A Multicentric Randomized Study,” Journal of Clinical Oncology, 19, 425–431.
   PubMed Web of Science ®Google Scholar
• Romano, J. P., Shaikh, A. M., and Wolf, M. (2014), “A Practical Twostep Method for Testing Moment Inequalities,” Econometrica, 82, 1979–2002.
   Web of Science ®Google Scholar
• Rosenbaum, P. R. (2002), Observational Studies, New York: Springer.
   Google Scholar
• Schochet, P. Z., Puma, M., and Deke, J., (2014), “Understanding Variation in Treatment Effects in Education Impact Evaluations: An Overview of Quantitative Methods,” Report no. NCEE 2014-4017, U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance.
   Google Scholar
• Šidák, Z. (1967), “Rectangular Confidence Regions for the Means of Multivariate Normal Distributions,” Journal of the American Statistical Association, 62, 626–633.
   Web of Science ®Google Scholar
• Simes, R. J. (1986), “An Improved Bonferroni Procedure for Multiple Tests of Significance,” Biometrika, 73, 751–754.
   Web of Science ®Google Scholar
• Sun, W., and McLain, A. C. (2012), “Multiple Testing of Composite Null Hypotheses in Heteroscedastic Models,” Journal of the American Statistical Association, 107, 673–687.
   Web of Science ®Google Scholar
• Viechtbauer, W. (2010), “Conducting Meta-Analyses in R with the Metafor Package,” Journal of Statistical Software, 36, 1–48.
   Web of Science ®Google Scholar
• Wang, R., Lagakos, S. W., Ware, J. H., Hunter, D. J., and Drazen, J. M. (2007), “Statistics in Medicinereporting of Subgroup Analyses in Clinical Trials,” New England Journal of Medicine, 357, 2189–2194.
   PubMed Web of Science ®Google Scholar
• Whitt, W. (1980), “Uniform Conditional Stochastic Order,” Journal of Applied Probability, 17, 112–123.
   Web of Science ®Google Scholar
• Wiens, B. L. (2003), “A Fixed Sequence Bonferroni Procedure for Testing Multiple Endpoints,” Pharmaceutical Statistics, 2, 211–215.
   Google Scholar
• Wright, S. P. (1992), “Adjusted p-Values for Simultaneous Inference,” Biometrics, 48, 1005–1013.
   Web of Science ®Google Scholar
• Yu, L., Suh, E.-Y., and Pan, G. (2014), QualInt: Test for Qualitative Interactions, R Package Version 1.0.0, available at https://CRAN.R-project.org/package=QualInt.
   Google Scholar
• Zaykin, D. V., Zhivotovsky, L. A., Westfall, P. H., and Weir, B. S. (2002), “Truncated Product Method for Combining p-Values,” Genetic Epidemiology, 22, 170–185.
   Google Scholar
• Zhao, Q. (2017), “On Sensitivity Value of Pair-Matched Observational Study,” Journal of the American Statistical Association, accepted.
   Google Scholar