Quantifying Replicability and Consistency in Systematic Reviews

References

• Aanagiotou, O. A., and Trikalinos, T. A. (2015), “On Effect Measures, Heterogeneity, and The Laws of Nature,” Epidemiology, 26, 710–713. DOI: 10.1097/EDE.0000000000000359.
   PubMed Web of Science ®Google Scholar
• Anzures-Cabrera, J., and Higgins, J. P. T. (2010), “Graphical Displays for Meta-analysis: An Overview with Suggestions for Practice,” Research Synthesis Methods, 1, 66–80. DOI: 10.1002/jrsm.6.
   PubMed Web of Science ®Google Scholar
• Bax, L., Yu, L. M., Ikeda, N., Tsuruta, H., and Moons, K. M. (2006), “Development and Validation of MIX: Comprehensive Free Software for Meta-Analysis of Causal Research Data,” BMC Medical Research Methodology, 6, 50. DOI: 10.1186/1471-2288-6-50.
   PubMedGoogle Scholar
• Benjamini, Y., and Heller, R. (2008), “Screening for Partial Conjunction Hypotheses,” Biometrics, 64, 1215–1222. DOI: 10.1111/j.1541-0420.2007.00984.x.
   PubMed Web of Science ®Google Scholar
• Benjamini, Y., Heller, R., and Yekutieli, D. (2009), “Selective Inference in Complex Research,” Philosophical Transactions of the Royal Society, Series A, 367, 4255–4271. DOI: 10.1098/rsta.2009.0127.
   Google Scholar
• Borenstein, M., Hedges, L. V., Higgins, J. P. T., and Rothstein, H. R. (2009), Introduction to Meta-Analysis, Chichester: Wiley.
   Google Scholar
• Borenstein, M., Higgins, J. P. T., Hedges, L. V., and Rothstein, H. R. (2017), “Basics of Meta-Analysis: I-2 is Not An Absolute Measure of Heterogeneity,” Research Synthesis Methods, 8, 5–18. DOI: 10.1002/jrsm.1230.
   PubMed Web of Science ®Google Scholar
• Collins, F. S., and Tabak, L. A. (2014), “NIH Plans to Enhance Reproducibility,” Nature, 505, 612–613. DOI: 10.1038/505612a.
   PubMed Web of Science ®Google Scholar
• Deeks, J. J., Higgins, J. P. T., Altman, D. G., and Cochrane Statistical Methods Group. (2019), “Analysing Data and Undertaking Meta-Analyses,” in Cochrane Handbook for Systematic Reviews of Interventions, eds. J. P. T. Higgins, J. Thomas, J. Chandler, M. Cumpston, T. Li, M. J. Page, and V. A. Welch, pp. 241–284, Hoboken, NJ: Wiley.
   Google Scholar
• Efron, B. (2012), Large-Scale Inference: Empirical Bayes Methods for Estimation, Testing, and Prediction (Vol. 1), Cambridge: Cambridge University Press.
   Google Scholar
• Fisher, R. A. (1935), The Design of Experiments (1st ed.), New York, NY: Hafner Press.
   Google Scholar
• Fisher, R. A. (1946), Statistical Methods for Research Workers (10th ed.), p. 354, Edinburgh, UK: Oliver and Boyd.
   Google Scholar
• Follmann, D. (1996), “A Simple Multivariate Test for One-Sided Alternatives,” Journal of the American Statistical Association, 91, 854–861. DOI: 10.1080/01621459.1996.10476953.
   Web of Science ®Google Scholar
• Frick, H. (1994), “A Maxmin Linear Test of Normal Means and Its Application to Lachin’s Data,” Communications in Statistics-Theory and Methods, 23, 1021–1029. DOI: 10.1080/03610929408831302.
   Web of Science ®Google Scholar
• Futschik, A., Taus, T., and Zehetmayer, S. (2019), “An Omnibus Test for The Global Null Hypothesis,” Statistical Methods in Medical Research, 28, 2292–2304. DOI: 10.1177/0962280218768326.
   PubMed Web of Science ®Google Scholar
• Gail, M., and Simon, R. (1985), “Testing for Qualitative Interactions Between Treatment Effects and Patient Subsets,” Biometrics, 41, 361–372. DOI: 10.2307/2530862.
   PubMed Web of Science ®Google Scholar
• Gaudino, M., Hameed, I., Farkouh, M. E., Rahouma, M., Naik, A., Robinson, N. B., Ruan, Y., Demetres, M., Biondi-Zoccai, G., Angiolillo, D. J., Bagiella, E., Charlson, M. E., Benedetto, U., Ruel, M., Taggart, D., Girardi, L. N., Bhatt, D. L., and Fremes, S. E. (2020), “Overall and Cause-Specific Mortality in Randomized Clinical Trials Comparing Percutaneous Interventions With Coronary Bypass Surgery: A Meta-analysis,” JAMA Internal Medicine, 180, 1638–1646. DOI: 10.1001/jamainternmed.2020.4748.
   PubMed Web of Science ®Google Scholar
• Gibson, E. W. (2021), “The Role of p-Values in Judging the Strength of Evidence and Realistic Replication Expectations,” Statistics in Biopharmaceutical Research, 13, 6–18. DOI: 10.1080/19466315.2020.1724560.
   Web of Science ®Google Scholar
• Heller, R. (2011), “Discussion of “Multiple Testing for Exploratory Research” by J.J. Goeman and A. Solari,” Statistical Science, 26, 598–600.
   Google Scholar
• Higgins, J. P. T., and Green, S., eds. (2011), Cochrane Handbook for Systematic Reviews of Interventions. Version 5.1.0 [updated March 2011], The Cochrane Collaboration. Available at www.cochrane-handbook.org.
   Google Scholar
• Higgins, J. P. T., and Thompson, S. G. (2002), “Quantifying Heterogeneity in A Meta-Analysis,” Statistics in Medicine, 21, 1539–1558. DOI: 10.1002/sim.1186.
   PubMed Web of Science ®Google Scholar
• Higgins, J. P. T., Thompson, S. G., and Spiegelhalter, D. J. (2009), “A Re-evaluation of Random-Effects Meta-Analysis,” Journal of The Royal Statistical Society, Series A, 172, 137–159. DOI: 10.1111/j.1467-985X.2008.00552.x.
   PubMed Web of Science ®Google Scholar
• Hsu, J. Y., Small, D. S., and Rosenbaum, P. R. (2013), “Effect Modification and Design Sensitivity in Observational Studies,” Journal of the American Statistical Association, 108, 135–148. DOI: 10.1080/01621459.2012.742018.
   Web of Science ®Google Scholar
• Hughes, D., Judge, C., Murphy, R., Loughlin, E., Costello, M., Whiteley, W., Bosch, J., O’Donnell, M., and Canavan, M. (2020), “Association of Blood Pressure Lowering With Incident Dementia or Cognitive Impairment: A Systematic Review and Meta-Analysis,” JAMA, 323, 1934–1944. DOI: 10.1001/jama.2020.4249.
   PubMed Web of Science ®Google Scholar
• Kafkafi, N., Benjamini, Y., Sakov, A., Elmer, G. I., and Golani, I. (2005), “Genotype-Environment Interactions in Mouse Behavior: A Way Out of The Problem,” Proceedings of the National Academy of Sciences of the United States of America, 102, 4619–4624. DOI: 10.1073/pnas.0409554102.
   PubMed Web of Science ®Google Scholar
• Kirsch, I., Deacon, B. J., Huedo-Medina, T. B., Scoboria, A., Moore, T. J., and Johnson, B. T. (2008), “Initial Severity and Antidepressant Benefits: A Meta-Analysis of Data Submitted to The Food and Drug Administration,” PLoS Medicine, 5, e45. DOI: 10.1371/journal.pmed.0050045.
   PubMed Web of Science ®Google Scholar
• Littell, R. C., and Folks, J. L. (1971), “Asymptotic Optimality of Fisher’s Method of Combining Independent Tests,” Journal of the American Statistical Association, 66, 802–806. DOI: 10.1080/01621459.1971.10482347.
   Web of Science ®Google Scholar
• Loughin, T. (2004), “A Systematic Comparison of Methods for Combining p-Values from Independent Tests,” Computational Statistics and Data Analysis, 47, 467–485. DOI: 10.1016/j.csda.2003.11.020.
   Web of Science ®Google Scholar
• Mathur, M. B., and VanderWeele, T. J. (2019), “New Metrics for Meta-Analyses of Heterogeneous Effects,” Statistics in Medicine, 38, 1336–1342. DOI: 10.1002/sim.8057.
   PubMed Web of Science ®Google Scholar
• Mathur, M. B., and VanderWeele, T. J. (2020a), “New Statistical Metrics for Multisite Replication Projects,” Journal of the Royal Statistical Society, Series A, 183, 1145–1166.
   Web of Science ®Google Scholar
• Mathur, M. B., and VanderWeele, T. J. (2020b), “Robust Metrics and Sensitivity Analyses for Meta-Analyses of Heterogeneous Effects,” Epidemiology (Cambridge, MA), 31, 356–358.
   PubMed Web of Science ®Google Scholar
• McNutt, M. (2014), “Journals Unite for Reproducibility,” Science, 346, 679. DOI: 10.1126/science.aaa1724.
   PubMed Web of Science ®Google Scholar
• MDI, Metered Dose Inhaler and Drug, Dry Powder Inhaler DPI (1998), “Guidance For Industry,” Center for Drug Evaluation and Research (CDER), 1000.
   Google Scholar
• National Academies of Sciences, Engineering, and Medicine (2019), Reproducibility and Replicability in Science, Washington DC: National Academies Press.
   Google Scholar
• Nuzzo, R. (2014), “Scientific Method: Statistical Errors,” Nature News, 506, 150–152. DOI: 10.1038/506150a.
   PubMed Web of Science ®Google Scholar
• Open Science Collaboration. (2015), “Estimating the Reproducibility of Psychological Science,” Science, 349, aac4716.
   PubMed Web of Science ®Google Scholar
• Owen, A. B. (2009), “Karl Pearson’s Meta-Analysis Revised,” Annals of Statistics, 37, 3867–3892.
   Web of Science ®Google Scholar
• Piantadosi, S., and Gail, M. (1993), “A Comparison of the Power of Two Tests for Qualitative Interactions,” Statistics in Medicine, 12, 1239–1248. DOI: 10.1002/sim.4780121305.
   PubMed Web of Science ®Google Scholar
• Pocock, S. J., Geller, N. L., and Tsiatis, A. A. (1987), “The Analysis of Multiple Endpoints in Clinical Trials,” Biometrics, 43, 487–498. DOI: 10.2307/2531989.
   PubMed Web of Science ®Google Scholar
• Riley, R. D., Higgins, J. P. T., and Deeks, J. J. (2011), “Interpretation of Random Effects Meta-Analyses,” British Medical Journal, 342, d549. DOI: 10.1136/bmj.d549.
   PubMed Web of Science ®Google Scholar
• Tang, D., Gnecco, C., and Geller, N. L. (1989), “An Approximate Likelihood Ratio Test for A Normal Mean Vector With Nonnegative Components with Application to Clinical Trials,” Biometrika, 76, 577–583. DOI: 10.1093/biomet/76.3.577.
   Web of Science ®Google Scholar
• Tarone, R. E. (1981), “On The Distribution of The Maximum of The Logrank Statistic and the Modified Wilcoxon Statistic,” Biometrics, 37, 79–85. DOI: 10.2307/2530524.
   Web of Science ®Google Scholar
• Turner, E. H., Matthews, A. M., Linardatos, E., Tell, R. A., and Rosenthal, R. (2008), “Selective Publication of Antidepressant Trials and its Influence on Apparent Efficacy,” New England Journal of Medicine, 358, 252–260. DOI: 10.1056/NEJMsa065779.
   PubMed Web of Science ®Google Scholar
• Turner, R. M., Jackson, D., Wei, Y., Thompson, S. G., and Higgins, J. P. T. (2015), “Predictive Distributions for Between-Study Heterogeneity and Simple Methods for their Application in Bayesian Meta-Analysis,” Statistics in Medicine, 34, 984–998. DOI: 10.1002/sim.6381.
   PubMed Web of Science ®Google Scholar
• Veroniki, A. A., Jackson, D., Viechtbauer, W., Bender, R., Bowden, J., Knapp, G., Kuss, O., Higgins, J. P. T., Langan, D., and Salanti, G. (2016), “Methods to Estimate the Between-Study Variance and its Uncertainty in Meta-Analysis,” Research Synthesis Methods, 7, 55–79. DOI: 10.1002/jrsm.1164.
   PubMed Web of Science ®Google Scholar
• Wang, J., and Owen, A. B. (2019), “Admissibility in Partial Conjunction Testing,” Journal of the American Statistical Association, 114, 158–168. DOI: 10.1080/01621459.2017.1385465.
   Web of Science ®Google Scholar
• Zaykin, D. V., Zhivotovsky, L. A., Westfall, P. H., and Weir, B. S. (2002), “Truncated Product Method for Combining p-Values,” Genetic Epidemiology, 22, 170–185. DOI: 10.1002/gepi.0042.
   PubMed Web of Science ®Google Scholar
• Zhang, J., Fu, H., and Carlin, B. P. (2015), “Detecting Outlying Trials in Network Meta-Analysis,” Statistics in Medicine, 34, 2695–2707. DOI: 10.1002/sim.6509.
   PubMed Web of Science ®Google Scholar
• Zhao, Q., Small, D. S., and Su, W. (2019), “Multiple Testing When Many p-Values are Uniformly Conservative, With Application to Testing Qualitative Interaction in Educational Interventions,” Journal of the American Statistical Association, 114, 1291–1304. DOI: 10.1080/01621459.2018.1497499.
   Web of Science ®Google Scholar