Why is Getting Rid of P-Values So Hard? Musings on Science and Statistics

References

• Baker, M. (2016), “1,500 Scientists Lift the Lid on Reproducibility,” Nature, 533, 452–454. DOI: 10.1038/533452a.
   PubMed Web of Science ®Google Scholar
• Batterham A.M., Hopkins W.G. (2006), “Making Meaningful Inferences About Magnitudes,” International Journal of Sports Physiological Performance, 1, 50–57. DOI: 10.1123/ijspp.1.1.50.
   PubMed Web of Science ®Google Scholar
• Benjamini, Y. (2016), “It’s Not the P-values’ Fault,” The American Statistician, ASA Statement supplementary materials, available at utas_a_1154108_sm5354.pdf
   Google Scholar
• Butson, M. (2018), “Will the Numbers Really Love You Back: Re-examining Magnitude-Based Inference,” [Internet] 2018, available at https://osf.io/yvj5r/.
   Google Scholar
• Camerer, C.F., Dreber, A., Holzmeister, F. Ho T-H., Huber J., Johannesson M., Kirchler M., Nave G., Nosek B.A., and Pfeiffer T. (2018), “Evaluating the Replicability of Social Science Experiments in Nature and Science Between 2010 and 2015,” Nature Human Behaviour, 2, 637–644. DOI: 10.1038/s41562-018-0399-z.
   PubMed Web of Science ®Google Scholar
• Cutler, S. J., Greenhouse, S. W., Cornfield, J., and Schneiderman, M. A. (1966), “The Role of Hypothesis Testing in Clinical Trials,” J Chron Disease, 19, 857–882.
   PubMedGoogle Scholar
• DeAngelis, C.D., Drazen, J. M., Frizelle, F. A., Haug, C., Hoey, J., Horton, R., Kotzin, S., Laine, C., Marusic, A., Overbeke, A. J., Schroeder, T. V., Sox, H. C., and Van Der Weyden, M.B. (2004), “Clinical Trial Registration: A Statement from the International Committee of Medical Journal Editors,” JAMA, 292, 1363–1364. DOI: 10.1001/jama.292.11.1363.
   PubMed Web of Science ®Google Scholar
• Echt, D.S., Liebson, P. R., Mitchell, L. B., Peters, R. W., Obias-Manno, D., Barker, A. H., Arensberg, D., Baker, A., Friedman, L., Greene, H. L., Huther, M. L., Richardson, D. W., and The Cast Investigators (1991), “Mortality and Morbidity in Patients Receiving Encainide, Flecaininde, or Placebo. The Cardiac Arrhythmia Suppression Trial,” The New England Journal of Medicine, 234, 12 781–788. DOI: 10.1056/NEJM199103213241201.
   Web of Science ®Google Scholar
• Food and Drug Administration, Guidance for the Use of Bayesian Statistics in Medical Device Clinical Trials Document (2010), available at https://www.fda.gov/downloads/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments/ucm071121.pdf
   Google Scholar
• Food and Drug Administration (2018), “Adaptive Designs for Clinical Trials of Drugs and Biologics Guidance for Industry — Draft Guidance,” available at https://www.fda.gov/downloads/drugs/guidances/ucm201790.pdf
   Google Scholar
• Gigerenzer, G, Swijtink, Z., Porter, T., Daston, L., Beatty, J., and Kruger, L. (1989), The Empire of Chance, Cambridge, UK: Cambridge University Press.
   Google Scholar
• Goligher, E.C., Tomlinson, G., Hajage, D. et al. (2018), “Extracorporeal Membrane Oxygenation for Severe Acute Respiratory Distress Syndrome and Posterior Probability of Mortality Benefit in a Post Hoc Bayesian Analysis of a Randomized Clinical Trial,” JAMA, 320, 2251–2259. DOI: 10.1001/jama.2018.14276.
   PubMed Web of Science ®Google Scholar
• Goodman, S. N. (1999a), “Towards Evidence-based Medical Statistics, I: The P-value Fallacy,” Annals of Internal Medicine, 130 995–1004. DOI: 10.7326/0003-4819-130-12-199906150-00008.
   PubMed Web of Science ®Google Scholar
• Goodman, S. N. (1999b), “Towards Evidence-based Medical Statistics, II: The Bayes Factor,” Annals of Internal Medicine, 130 1005–1013. DOI: 10.7326/0003-4819-130-12-199906150-00019.
   PubMed Web of Science ®Google Scholar
• Goodman, S. N. (2016) “The Next Questions: Who, What, When, Where and Why?” The American Statistician, 73(S1), this issue.
   Google Scholar
• Goodman, S. N. (2018), “How Confident Are You in a Research Finding? Say It with a Number!” Nature, 564, 7. DOI: 10.1038/d41586-018-07589-2.
   PubMed Web of Science ®Google Scholar
• Institute of Medicine (2012), Committee, on the Review of Omics-Based Tests for Predicting Patient Outcomes in Clinical Trials, ‘Evolution of Translational Omics’.
   Google Scholar
• Ioannidis, J. P., Fanelli, D., Dunne, D. D. and Goodman, S. N. (2015), Meta-research: Evaluation and Improvement of Research Methods and Practices,” PLoS Biology, 13 (10), e1002264. DOI: 10.1371/journal.pbio.1002264.
   PubMed Web of Science ®Google Scholar
• Kaplan, R. M., and Irvin, V. L. (2015), “Likelihood of Null Effects of Large NHLBI Clinical Trials Has Increased over Time,” PLoS One, 10, e0132382. DOI: 10.1371/journal.pone.0132382.
   PubMed Web of Science ®Google Scholar
• Kass, R. E., and Raftery, A. E. (1995), “Bayes Factors,” JASA, 90, 773–795. DOI: 10.1080/01621459.1995.10476572.
   Web of Science ®Google Scholar
• Laptook, A. R., Shankaran, S., Tyson, J. E., Munoz, B., Bell, E. F., Goldberg R. N., Parikh, N. A., Ambalavanan, N., Pedroza, C., Pappas, A., Das, A., Chaudhary, A.S., Ehrenkranz, R. A., Hensman, A. M., Van Meurs, K. P., Chalak, L. F., Khan, A. M., Hamrick, S. E. G., Sokol, G. M., Walsh, M. C., Poindexter, B. B., Faix, R. G., Watterberg, K. L., Frantz, I. D., Guillet, R., Devaskar, U., Truog, W. E., Chock, V. Y., Wyckoff, M. H., McGowan, E. C., Carlton, D. P., Harmon, H. M., Brumbaugh, J. E., Cotton, C. M., Sánchez, P. J., Hibbs, A. M., and Higgins, R. D. (2017), “Effect of Therapeutic Hypothermia Initiated After 6 Hours of Age on Death or Disability Among Newborns With Hypoxic-Ischemic Encephalopathy: A Randomized Clinical Trial,” JAMA, 318 (16), 1550–1560. DOI: 10.1001/jama.2017.14972.
   PubMed Web of Science ®Google Scholar
• Lewis, R. J., and Angus, D. C. (2018), “Time for Clinicians to Embrace Their Inner Bayesian?: Reanalysis of Results of a Clinical Trial of Extracorporeal Membrane Oxygenation,” JAMA, 320(21), 2208–2210. DOI: 10.1001/jama.2018.16916.
   PubMed Web of Science ®Google Scholar
• Mayo-Wilson, E., Vander Ley, K., Dickersin, K., Helfand, M. (2017), “Patient-Centered Outcomes Research Institute (PCORI) Methodology Standards to Improve the Design and Reporting of Research.” [Abstract] International Congress on Peer Review and Publication. https://peerreviewcongress.org/prc17-0326.
   Google Scholar
• Micheel C. M., Nass S. J., Omenn G. S. (2012), Evolution of Translational Omics: Lessons Learned and the Path Forward, National Academies Press; Washington, DC.
   Google Scholar
• Moher, D., Naudet, F., Cristea, I. A., Miedema, F., Ioannidis, J. P. A., and Goodman, S. N. (2018), “Assessing Scientists for Hiring, Promotion, and Tenure,” PLoS Biol. 16:e2004089. DOI: 10.1371/journal.pbio.2004089.
   PubMed Web of Science ®Google Scholar
• National Academies of Sciences, Engineering, and Medicine (2017), “Fostering Integrity in Research,” Washington, DC.
   Google Scholar
• PCORI Methodology Committee (2018), “PCORI Methodology Report.” available at https://www.pcori.org/research-results/about-our-research/research-methodology/pcori-methodology-report.
   Google Scholar
• Pearson, E.S. (1962), “Some Thoughts on Statistical Inference,” Annals of Mathematical Statistics, 33, 394–403. DOI: 10.1214/aoms/1177704566.
   Google Scholar
• Piller, C. (2015), ‘Failure to Report: A STAT Investigation of Clinical Trials’, Stat News, available at https://www.statnews.com/2015/12/13/clinical-trials-investigation/
   Google Scholar
• Porter, T. (1995), Trust In Numbers: The Pursuit Of Objectivity in Science and Public Life. Princeton: Princeton University Press.
   Google Scholar
• Quintana, M., Viele, K., and Lewis, R. J. (2017), “Bayesian Analysis: Using Prior Information to Interpret the Results of Clinical Trials,” JAMA, 318 (16), 1605–1606. DOI: 10.1001/jama.2017.15574.
   PubMed Web of Science ®Google Scholar
• Sainani K. L. (2018), “The Problem with Magnitude-based Inference,” Medicine and Science in Sports and Exercise, 50 (10), 2166–2176. DOI: 10.1249/MSS.0000000000001645.
   PubMed Web of Science ®Google Scholar
• Van Noorden, R. (2014), “Science Joins Push to Screen Statistics in Papers,” Nature, available at DOI: 10.1038/nature.2014.15509.
   Google Scholar
• Wang, M.Q., Yan, A. F., and Katz, R. V. (2018), “Researcher Requests for Inappropriate Analysis and Reporting: A U.S. Survey of Consulting Biostatisticians,” Annals of Internal Medicine, 169 (8), 554–558. DOI: 10.7326/M18-1230.
   PubMed Web of Science ®Google Scholar
• Wasserstein, R. L., and Lazar, N. A. (2016), “The ASA’s Statement on p-values: Context, Process, and Purpose,” The American Statistician, 70, 129–133. DOI: 10.1080/00031305.2016.1154108.
   Web of Science ®Google Scholar
• Yong, E. (2012), “A Failed Replication Draws a Scathing Personal Attack from a Psychology Professor,” Discover Magazine, available at https://bit.ly/2qc5bkz.
   Google Scholar
• Zarin, D. A., Tse, T., Williams, R. J., and Rajakannan, T. (2017), “Update on Trial Registration 11 Years after the ICMJE Policy Was Established,” The New England Journal of Medicine, 376 (4), 383–391. DOI: 10.1056/NEJMsr1601330.
   PubMed Web of Science ®Google Scholar