Advanced search
Publication Cover
The American Statistician Volume 73, 2019 - Issue sup1: Statistical Inference in the 21st Century: A World Beyond p < 0.05
Submit an article Journal homepage
36,764
Views
99
CrossRef citations to date
0
Altmetric
Getting to a Post “p<0.05” Era

What Have We (Not) Learnt from Millions of Scientific Papers with P Values?

John P. A. IoannidisDepartments of Medicine, of Health Research and Policy, of Biomedical Data Science, and of Statistics, Stanford University and Meta-Research Innovation Center at Stanford (METRICS), Stanford, CA
Pages 20-25 | Received 25 Nov 2017, Published online: 20 Mar 2019

Citations (99)

Keep up to date with the latest research on this topic with citation updates for this article.

Subscribe to citation updates

Read on this site (5)

Joshua Habiger & Ye Liang. (2022) Publication Policies for Replicable Research and the Community-Wide False Discovery Rate. The American Statistician 76:2, pages 131-141.
Read now
Venurs Loh, Shannon Sahlqvist, Jenny Veitch, Alison Carver, Ana Maria Contardo Ayala, Rachel Cole & Anna Timperio. (2022) Substituting passive for active travel—what is the potential among adolescents?. International Journal of Sustainable Transportation 16:1, pages 84-93.
Read now
Thomas Gwise, Mark D. Rothmann, H.M. James Hung, Anup Amatya, Rebecca Rothwell, Sue Jane Wang, Yute Wu, Fraser Smith, Yu-Ting Weng, Eugenio Andraca-Carrera, Stella Grosser, Somesh Chattopadhyay & Sylva H. Collins. (2021) Statement on P-values. Statistics in Biopharmaceutical Research 13:1, pages 57-58.
Read now
Ronald L. Wasserstein, Allen L. Schirm & Nicole A. Lazar. (2019) Moving to a World Beyond “p < 0.05”. The American Statistician 73:sup1, pages 1-19.
Read now
Maurits K.A. van Selms & Frank Lobbezoo. The reports of specific waking-state oral behaviours, including awake bruxism activities, and psychological distress have a dose-response relationship: A retrospective medical record study. CRANIO® 0:0, pages 1-12.
Read now

Articles from other publishers (94)

Patricia Wonch Hill, Grace M. Kelly, Julia McQuillan, Jorge Ledesma, Miranda Melson & G. Robin Gauthier. (2024) Exploring the Associations of Afterschool Science Participation and Friendships with Science Identities. Research in Science Education.
Crossref
Lucas Wright, Roxana Mika Muenster, Briana Vecchione, Tianyao Qu, Pika (Senhuang) Cai, Alan Smith, Comm 2450 Student Investigators, Jacob Metcalf & J. Nathan Matias. (2024) Null Compliance: NYC Local Law 144 and the challenges of algorithm accountability. Null Compliance: NYC Local Law 144 and the challenges of algorithm accountability.
Ahtisham Younas. (2024) Beyond ‘statistical significance’: A nontechnical primer of Bayesian statistics and Bayes factors for health researchers. Journal of Evaluation in Clinical Practice.
Crossref
Ahtisham Younas, Hussan Zeb, Angela Durante & Ercole Vellone. (2024) Sex based differences in depression, anxiety, and quality of life and predictors of quality of life among South Asian individuals with chronic obstructive pulmonary disease: A Bayesian analysis. Social Science & Medicine 351, pages 116989.
Crossref
Ben G. Fitzpatrick, Dennis M. Gorman & Caitlin Trombatore. (2024) Impact of redefining statistical significance on P-hacking and false positive rates: An agent-based model. PLOS ONE 19:5, pages e0303262.
Crossref
Raymand Pang, Christopher Meehan, George Maple, Georgina Norris, Ellie Campbell, Katie Tucker, Alison Mintoft, Francisco Torrealdea, Alan Bainbridge, Mariya Hristova, John Barks, Xavier Golay, Joseph Standing & Nicola J. Robertson. (2024) Melatonin reduces brain injury following inflammation‐amplified hypoxia–ischemia in a translational newborn piglet study of neonatal encephalopathy. Journal of Pineal Research 76:4.
Crossref
Wendy M. Novicoff. (2024) CORR Insights®: Is Operative Time Associated With Obesity-related Outcomes in TKA?. Clinical Orthopaedics & Related Research 482:5, pages 810-813.
Crossref
Dirk Luthin. (2024) Osteopathie und evidenzbasierte Medizin, Teil 2: Kritik an der EBM. DO - Deutsche Zeitschrift für Osteopathie 22:02, pages 34-41.
Crossref
Joost de Winter. (2024) Can ChatGPT be used to predict citation counts, readership, and social media interaction? An exploration among 2222 scientific abstracts. Scientometrics 129:4, pages 2469-2487.
Crossref
Martin C. Refalo, Jacob F. Remmert, Joshua C. Pelland, Zac P. Robinson, Michael C. Zourdos, D. Lee HamiltonJackson J. FyfeEric R. Helms. (2024) Accuracy of Intraset Repetitions-in-Reserve Predictions During the Bench Press Exercise in Resistance-Trained Male and Female Subjects. Journal of Strength & Conditioning Research 38:3, pages e78-e85.
Crossref
Todd Mitton. (2024) Economic Significance in Corporate Finance. The Review of Corporate Finance Studies 13:1, pages 38-79.
Crossref
Claudia Angelini. 2024. Reference Module in Life Sciences. Reference Module in Life Sciences.
Reinout HeijungsReinout Heijungs. 2024. Probability, Statistics and Life Cycle Assessment. Probability, Statistics and Life Cycle Assessment 233 330 .
Reinout HeijungsReinout Heijungs. 2024. Probability, Statistics and Life Cycle Assessment. Probability, Statistics and Life Cycle Assessment 789 914 .
Michael Meier & Christof Bigler. (2023) Process-oriented models of autumn leaf phenology: ways to sound calibration and implications of uncertain projections. Geoscientific Model Development 16:23, pages 7171-7201.
Crossref
Fanny Pandolfi, Christian Brun-Buisson, Didier Guillemot & Laurence Watier. (2023) Care pathways of sepsis survivors: sequelae, mortality and use of healthcare services in France, 2015–2018. Critical Care 27:1.
Crossref
Dennis M. Gorman. (2023) Commentary: Effectiveness of a hybrid digital substance abuse prevention approach combining e-learning and in-person class sessions. Frontiers in Digital Health 5.
Crossref
Fabio Parisi, Claudio Feliciani & Ruggiero Lovreglio. (2023) What do we head for while exiting a room? a novel parametric distance map for pedestrian dynamic simulations. Transportation Research Part C: Emerging Technologies 156, pages 104335.
Crossref
Olivier Supplisson & Mircea T. Sofonea. (2023) Pièges et mésusages en analyse de données. Anesthésie & Réanimation 9:5-6, pages 440-450.
Crossref
George Gunnesch-Luca, Karsten I. Paul & Klaus Moser. (2023) A Bayesian Regression Analysis of the Effects of Narcissism on Group Performance. Journal of Personnel Psychology 22:4, pages 215-225.
Crossref
Katherine Christian, Jo-ann Larkins & Michael R. Doran. (2023) We must improve conditions and options for Australian ECRs. Nature Human Behaviour 7:7, pages 1038-1041.
Crossref
Olimpio Montero, Mikael Hedeland & David Balgoma. (2023) Trials and tribulations of statistical significance in biochemistry and omics. Trends in Biochemical Sciences 48:6, pages 503-512.
Crossref
Yizhao Li, Paris Vakiel, Kevin Adanty, Simon Ouellet, Albert H. Vette, Donald Raboud & Christopher R. Dennison. (2023) Influence of surrogate scalp material and thickness on head impact responses: Toward a biofidelic head-brain physical model. Journal of the Mechanical Behavior of Biomedical Materials 142, pages 105859.
Crossref
Oliwier Dziadkowiec. (2023) Alternatives to Reporting Statistically and Clinically Meaningful Results. Journal of Obstetric, Gynecologic & Neonatal Nursing 52:3, pages 178-181.
Crossref
Joel S. Milner, Julie L. Crouch, Joe Ammar & Audra Jensen. (2023) Effect sizes and effect size benchmarks in family violence research. Child Abuse & Neglect 139, pages 106095.
Crossref
Michael C Robitaille, Jeff M Byers, Joseph A Christodoulides & Marc P Raphael. (2023) Automated cell segmentation for reproducibility in bioimage analysis. Synthetic Biology 8:1.
Crossref
Joshua J. Reynolds. (2021) Let’s Talk about Stats: Revising Our Approach to Teaching Statistics in Psychology. Psychological Reports 126:1, pages 5-33.
Crossref
Nathaniel J Himmel, Akira Sakurai, Atit A Patel, Shatabdi Bhattacharjee, Jamin M Letcher, Maggie N Benson, Thomas R Gray, Gennady S Cymbalyuk & Daniel N Cox. (2023) Chloride-dependent mechanisms of multimodal sensory discrimination and nociceptive sensitization in Drosophila. eLife 12.
Crossref
Rohit Goswami, Ruhila S, Amrita Goswami, Sonaly Goswami & Debabrata Goswami. 2023. Simulated Annealing - A Flexible Metaheuristic That Can Be Learned by Example [Cancelled]. Simulated Annealing - A Flexible Metaheuristic That Can Be Learned by Example [Cancelled].
Cristian Mesquida, Jennifer Murphy, Daniël Lakens & Joe Warne. (2022) Replication concerns in sports and exercise science: a narrative review of selected methodological issues in the field. Royal Society Open Science 9:12.
Crossref
David Johnstone. (2022) Accounting research and the significance test crisis. Critical Perspectives on Accounting 89, pages 102296.
Crossref
Fanny Pandolfi, Christian Brun-Buisson, Didier Guillemot & Laurence Watier. (2022) One-year hospital readmission for recurrent sepsis: associated risk factors and impact on 1-year mortality—a French nationwide study. Critical Care 26:1.
Crossref
Christine C Guo, Patrizia Andrea Chiesa, Carl de Moor, Mir Sohail Fazeli, Thomas Schofield, Kimberly Hofer, Shibeshih Belachew & Alf Scotland. (2022) Digital Devices for Assessing Motor Functions in Mobility-Impaired and Healthy Populations: Systematic Literature Review. Journal of Medical Internet Research 24:11, pages e37683.
Crossref
Joshua J. Reynolds. (2021) Improving the Assessment of Teaching Effectiveness With the Nonequivalent Dependent Variables Approach. Teaching of Psychology 49:4, pages 381-387.
Crossref
Oskar Hougaard Jefsen, Yury Shtyrov, Kit Melissa Larsen & Martin J. Dietz. (2022) The 40-Hz auditory steady-state response in bipolar disorder: A meta-analysis. Clinical Neurophysiology 141, pages 53-61.
Crossref
Robert Schleusener. (2022) Osteopathische Medizin – Wo bleibt der Beweis?. Osteopathische Medizin 23:3, pages 13-19.
Crossref
Sarah Nostedt & Ari R Joffe. (2022) Critical Care Randomized Trials Demonstrate Power Failure: A Low Positive Predictive Value of Findings in the Critical Care Research Field. Journal of Intensive Care Medicine 37:8, pages 1082-1093.
Crossref
Adrian Erasmus, Bennett Holman & John P A Ioannidis. (2022) Data-dredging bias. BMJ Evidence-Based Medicine 27:4, pages 209-211.
Crossref
Fang Wang, Jiajie Yin, Lin Hu, Mingliang Wang, Xin Liu, Karol Miller & Adam Wittek. (2022) Should anthropometric differences between the commonly used pedestrian computational biomechanics models and Chinese population be taken into account when predicting pedestrian head kinematics and injury in vehicle collisions in China?. Accident Analysis & Prevention 173, pages 106718.
Crossref
Matthew Bennett & Emma GoodallMatthew Bennett & Emma Goodall. 2022. Addressing Underserved Populations in Autism Spectrum Research. Addressing Underserved Populations in Autism Spectrum Research 177 214 .
Rehana K. Leak & James B. Schreiber. (2022) Mechanistic Research for the Student or Educator (Part I of II). Frontiers in Pharmacology 13.
Crossref
Ron S. Kenett & Bernard G. Francq. (2022) Helping reviewers assess statistical analysis: A case study from analytic methods. Analytical Science Advances 3:5-6, pages 212-222.
Crossref
Simone Di Plinio. (2022) Testing the Magnitude of Correlations Across Experimental Conditions. Frontiers in Psychology 13.
Crossref
Xiao Liu, Zhiyong Zhang & Lijuan Wang. (2022) Bayesian hypothesis testing of mediation: Methods and the impact of prior odds specifications. Behavior Research Methods 55:3, pages 1108-1120.
Crossref
Fanny Pandolfi, Didier Guillemot, Laurence Watier & Christian Brun-Buisson. (2022) Trends in bacterial sepsis incidence and mortality in France between 2015 and 2019 based on National Health Data System (Système National des données de Santé (SNDS)): a retrospective observational study. BMJ Open 12:5, pages e058205.
Crossref
Mark L. Taper, José M. Ponciano & Yukihiko Toquenaga. (2022) Editorial: Evidential Statistics, Model Identification, and Science. Frontiers in Ecology and Evolution 10.
Crossref
Ulrich Dirnagl, Georg N. Duda, David W. Grainger, Petra Reinke & Ronenn Roubenoff. (2022) Reproducibility, relevance and reliability as barriers to efficient and credible biomedical technology translation. Advanced Drug Delivery Reviews 182, pages 114118.
Crossref
Frank A. Bosco. (2022) Accumulating Knowledge in the Organizational Sciences. Annual Review of Organizational Psychology and Organizational Behavior 9:1, pages 441-464.
Crossref
Christian Salas-Eljatib, Joaquín Riquelme-Alarcón, Pablo J. Donoso, Diego Ponce & Daniel P. Soto. (2022) Analysing changes in spatial point patterns: A proposal using data from a forest thinning experiment. Forest Ecosystems 9, pages 100081.
Crossref
Nori Geary & Suzanne Higgs. (2022) Guidelines on design, measurement and statistics for Appetite. Appetite 168, pages 105731.
Crossref
Venera Tomaselli, Giulio Giacomo Cantone & Vincenzo Miracula. 2022. Studies in Theoretical and Applied Statistics. Studies in Theoretical and Applied Statistics 369 392 .
Ruslan Masharipov, Irina Knyazeva, Yaroslav Nikolaev, Alexander Korotkov, Michael Didur, Denis Cherednichenko & Maxim Kireev. (2021) Providing Evidence for the Null Hypothesis in Functional Magnetic Resonance Imaging Using Group-Level Bayesian Inference. Frontiers in Neuroinformatics 15.
Crossref
Agustín Ciapponi, José M. Belizán, Gilda Piaggio & Sanni Yaya. (2021) There is life beyond the statistical significance. Reproductive Health 18:1.
Crossref
Lauri Raittio, Antti Launonen, Ville M. Mattila & Aleksi Reito. (2021) Estimates of the mean difference in orthopaedic randomized trials: obligatory yet obscure. BMC Medical Research Methodology 21:1.
Crossref
Dirk Stengel, Wolf Mutschler, Luzi Dubs, Stephan Kirschner & Tobias Renkawitz. (2021) Klinische Studien in Unfallchirurgie und Orthopädie: lesen, interpretieren und umsetzenClinical studies in trauma surgery and orthopedics: read, interpret and implement. Der Unfallchirurg 124:12, pages 1007-1017.
Crossref
Chiyo Ito, Atsushi Hashimoto, Kohei Uemura & Koji Oba. (2021) Misleading Reporting (Spin) in Noninferiority Randomized Clinical Trials in Oncology With Statistically Not Significant Results. JAMA Network Open 4:12, pages e2135765.
Crossref
Brinley N. Zabriskie, Chris Corcoran & Pralay Senchaudhuri. (2021) A permutation‐based approach for heterogeneous meta‐analyses of rare events. Statistics in Medicine 40:25, pages 5587-5604.
Crossref
Xiongbin Kang, Dong Mei Deng, Wim Crielaard & Bernd W. Brandt. (2021) Reprocessing 16S rRNA Gene Amplicon Sequencing Studies: (Meta)Data Issues, Robustness, and Reproducibility. Frontiers in Cellular and Infection Microbiology 11.
Crossref
Riko Kelter. (2021) Bayesian Hodges-Lehmann tests for statistical equivalence in the two-sample setting: Power analysis, type I error rates and equivalence boundary selection in biomedical research. BMC Medical Research Methodology 21:1.
Crossref
Nathaniel J. Himmel, Jamin M. Letcher, Akira Sakurai, Thomas R. Gray, Maggie N. Benson, Kevin J. Donaldson & Daniel N. Cox. (2021) Identification of a neural basis for cold acclimation in Drosophila larvae. iScience 24:6, pages 102657.
Crossref
Juan Manuel Völker, Federico Gabriel Arguissain, Ole Kæseler Andersen & José Biurrun Manresa. (2021) Variability and effect sizes of intracranial current source density estimations during pain: Systematic review, experimental findings, and future perspectives. Human Brain Mapping 42:8, pages 2461-2476.
Crossref
Carl P. Walther. (2021) An impending obituary for the primacy of P values in glomerulonephritis trial results?. Kidney International 99:5, pages 1241-1242.
Crossref
Nathalie C. Støer, Edoardo Botteri, G. Hege Thoresen, Øystein Karlstad, Elisabete Weiderpass, Søren Friis, Anton Pottegård & Bettina K. Andreassen. (2021) Drug Use and Cancer Risk: A Drug-Wide Association Study (DWAS) in Norway. Cancer Epidemiology, Biomarkers & Prevention 30:4, pages 682-689.
Crossref
Lifeng Lin. (2020) Factors that impact fragility index and their visualizations. Journal of Evaluation in Clinical Practice 27:2, pages 356-364.
Crossref
David W. Moreau, Hakan Atakisi & Robert E. Thorne. (2021) Ice in biomolecular cryocrystallography. Acta Crystallographica Section D Structural Biology 77:4, pages 540-554.
Crossref
Andreas Kleppe, Ole-Johan Skrede, Sepp De Raedt, Knut Liestøl, David J. Kerr & Håvard E. Danielsen. (2021) Designing deep learning studies in cancer diagnostics. Nature Reviews Cancer 21:3, pages 199-211.
Crossref
Reinout Heijungs. (2021) Selecting the best product alternative in a sea of uncertainty. The International Journal of Life Cycle Assessment 26:3, pages 616-632.
Crossref
Steven A. Frost, Evan Alexandrou, Luis Schulz & Anders Aneman. (2020) Interpreting the results of clinical trials, embracing uncertainty: A Bayesian approach. Acta Anaesthesiologica Scandinavica 65:2, pages 146-150.
Crossref
Riko Kelter. (2021) On Two Measure-Theoretic Aspects of the Full Bayesian Significance Test for Precise Bayesian Hypothesis Testing †. On Two Measure-Theoretic Aspects of the Full Bayesian Significance Test for Precise Bayesian Hypothesis Testing †.
Anna Carolina Miola & Hélio Amante Miot. (2021) P-valor e dimensão do efeito em estudos clínicos e experimentais. Jornal Vascular Brasileiro 20.
Crossref
Dolores Frias-Navarro, Marcos Pascual-Soler, José Perezgonzalez, Héctor Monterde-i-Bort & Juan Pascual-Llobell. (2021) Spanish Scientists’ Opinion about Science and Researcher Behavior. The Spanish Journal of Psychology 24.
Crossref
Durga Prasanna Misra, Olena Zimba & Armen Yuri Gasparyan. (2020) Statistical data presentation: a primer for rheumatology researchers. Rheumatology International 41:1, pages 43-55.
Crossref
MaryLena Bleile. 2021. Intelligent Computing. Intelligent Computing 196 209 .
Arvid Erlandsson, Mattias Wingren & Per A. Andersson. (2020) Type and amount of help as predictors for impression of helpers. PLOS ONE 15:12, pages e0243808.
Crossref
Riko Kelter. (2020) Bayesian alternatives to null hypothesis significance testing in biomedical research: a non-technical introduction to Bayesian inference with JASP. BMC Medical Research Methodology 20:1.
Crossref
Riko Kelter. (2020) Analysis of Bayesian posterior significance and effect size indices for the two-sample t-test to support reproducible medical research. BMC Medical Research Methodology 20:1.
Crossref
Marianne Huebner, Werner Vach, Saskia le Cessie, Carsten Oliver Schmidt & Lara Lusa. (2020) Hidden analyses: a review of reporting practice and recommendations for more transparent reporting of initial data analyses. BMC Medical Research Methodology 20:1.
Crossref
Marc Sorigue, Eloi Cañamero & Juan‐Manuel Sancho. (2020) Precision medicine in follicular lymphoma: Focus on predictive biomarkers. Hematological Oncology 38:5, pages 625-639.
Crossref
Todd Schneberk, Brian Raffetto, Joseph Friedman, Andrew Wilson, David Kim & David L. Schriger. (2020) Opioid prescription patterns among patients who doctor shop; Implications for providers. PLOS ONE 15:5, pages e0232533.
Crossref
Marc Sorigue, Andrea Espasa, Lurdes Zamora & Jordi Junca. (2020) Residual disease by flow cytometry in patients with nucleophosmin-mutated acute myeloblastic leukemia. Annals of Hematology 99:5, pages 1135-1136.
Crossref
Claus Niederau. (2020) Schicke die statistische Signifikanz in den Ruhestand!. Gastro-News 7:2, pages 10-12.
Crossref
Mariana RP Alves. (2020) The Natural Fallacy in a Post‐Truth era. EMBO reports 21:2.
Crossref
Caridad Pontes, Corinne Zara, Josep Torrent-Farnell, Merce Obach, Cristina Nadal, Patricia Vella-Bonanno, Michael Ermisch, Steven Simoens, Renata Curi Hauegen, Jolanta Gulbinovic, Angela Timoney, Antony P. Martin, Tanja Mueller, Anna Nachtnebel, Stephen Campbell, Gisbert Selke, Tomasz Bochenek, Celia C. Rothe, Ileana Mardare, Marion Bennie, Jurij Fürst, Rickard E. Malmstrom & Brian Godman. (2019) Time to Review Authorisation and Funding for New Cancer Medicines in Europe? Inferences from the Case of Olaratumab. Applied Health Economics and Health Policy 18:1, pages 5-16.
Crossref
Minas Sifakis, Michael N. Kalochristianakis, Julieta G. García-Donas, Oguzhan Ekizoglu & Elena F. Kranioti. 2020. Statistics and Probability in Forensic Anthropology. Statistics and Probability in Forensic Anthropology 109 124 .
Eric P. Smith. (2019) Ending Reliance on Statistical Significance Will Improve Environmental Inference and Communication. Estuaries and Coasts 43:1, pages 1-6.
Crossref
Henning Tiemeier & Ryan Muetzel. 2020. Mental Health and Illness of Children and Adolescents. Mental Health and Illness of Children and Adolescents 117 138 .
Henning Tiemeier & Ryan Muetzel. 2020. Mental Health and Illness of Children and Adolescents. Mental Health and Illness of Children and Adolescents 1 22 .
Roberta W. Scherer & Ian J. Saldanha. (2019) How should systematic reviewers handle conference abstracts? A view from the trenches. Systematic Reviews 8:1.
Crossref
Vincent S. Staggs. (2019) Pervasive errors in hypothesis testing: Toward better statistical practice in nursing research. International Journal of Nursing Studies 98, pages 87-93.
Crossref
Alexander Koplenig. (2019) A non-parametric significance test to compare corpora. PLOS ONE 14:9, pages e0222703.
Crossref
Ulrich Gunter, Irem Önder & Egon Smeral. (2019) Scientific value of econometric tourism demand studies. Annals of Tourism Research 78, pages 102738.
Crossref
Todd Mitton. (2020) Economic Significance in Corporate Finance. SSRN Electronic Journal.
Crossref
Ron S. Kenett. (2020) Reviewing of Applied Research with an Industry 4.0 Perspective. SSRN Electronic Journal.
Crossref
John L. Kmetz. (2003) The Skeptic's Handbook: Consumer Guidelines and a Critical Assessment of Business and Management Research. SSRN Electronic Journal.
Crossref

Related research

People also read lists articles that other readers of this article have read.

Recommended articles lists articles that we recommend and is powered by our AI driven recommendation engine.

Cited by lists all citing articles based on Crossref citations.
Articles with the Crossref icon will open in a new tab.