A Proposed Hybrid Effect Size Plus p-Value Criterion: Empirical Evidence Supporting its Use

References

• Benjamin, D.J., Berger, J., Johannesson, M., Nosek, B. A., Wagenmakers, E.-J., Berk, R., …, Johnson, V. (2017, July 22). “Redefine Statistical Significance,” PsyArXiv Preprints (Online), available at https://psyarxiv.com/mky9j
   Google Scholar
• Berger, J.O., and Delampady, M. (1987), “Testing Precise Hypotheses,” Statistical Science, 2(3), 317–352. DOI: 10.1214/ss/1177013238.
   Google Scholar
• Berger, J.O., and Sellke, T. (1987), “Rejoinder to Comments on ‘Testing a Point Null Hypothesis: The Irreconcilability of p Values and Evidence,”’ Journal of the American Statistical Association, 82(397), 135–139. DOI: 10.2307/2289139.
   Web of Science ®Google Scholar
• Blume J.D., D’Agostino McGowan, L., Dupont W.D., Greevy, R.A. Jr (2018) “Second-Generation pValues: Improved Rigor, Reproducibility, & Transparency in Statistical Analyses,” PLoS ONE, 13(3): e0188299, available at DOI: 10.1371/journal.pone.0188299.
   PubMed Web of Science ®Google Scholar
• Chow, S.L. (1988), “Significance Test or Effect size?” Psychological Bulletin, 103, 105–110. DOI: 10.1037/0033-2909.103.1.105.
   Web of Science ®Google Scholar
• Cohen, J. (1988), Statistical Power Analysis for the Behavioral Sciences (2nd ed.), Hillsdale, NJ: Lawrence Erlbaum Associates.
   Google Scholar
• Colquhoun, D. (2014), “An Investigation of the False Discovery Rate and the Misinterpretation of p-values,” Royal Society Open Science, 1: 140216. DOI: 10.1098/rsos.140216.
   PubMed Web of Science ®Google Scholar
• Ellis, P.D. (2010), The Essential Guide to Effect Sizes: Statistical Power, Meta-Analysis, and the Interpretation of Research Results, Cambridge, UK: Cambridge University Press.
   Google Scholar
• Fisher, R. A. (1925), Statistical Methods for Research Workers (1st ed.), Edinburgh, Scotland: Oliver & Boyd. [Chapter 3, 5th paragraph.]
   Google Scholar
• Folger, R. (1989), “Significance Tests and the Duplicity of Binary Decisions,” Psychological Bulletin, 106, 155–160. DOI: 10.1037/0033-2909.106.1.155.
   Web of Science ®Google Scholar
• Goodman, W.M. (2010), “The Undetectable Difference: An Experimental Look at the ‘Problem’ of p-Values,” in American Statistical Association JSM Proceedings, available at http://www.statlit.org/pdf/2010GoodmanASA.pdf.
   Google Scholar
• Greenland, S. (2017). “Invited Commentary: The need for Cognitive Science in Methodology,” American Journal of Epidemiology, 186, 639–645. DOI: 10.1093/aje/kwx259.
   PubMed Web of Science ®Google Scholar
• Howard, G.S., Maxwell, S.E., and Fleming, K.J. (2000), “The Proof of the Pudding: An Illustration of the Relative Strengths of Null Hypothesis, Meta-Analysis, and Bayesian Analysis,” Psychological Methods, 5(3), 315–332. DOI: 10.1037/1082-989X.5.3.315.
   PubMed Web of Science ®Google Scholar
• Krueger, J.I., and Heck, P. R. (2017), “The Heuristic Value of p in Inductive Statistical Inference,” Frontiers in Psychology, available at http://journal.frontiersin.org/article/10.3389/fpsyg.2017.00908/abstract
   Google Scholar
• Lakens, D., Adolfi, F. G., Albers, C. J., Anvari, F., Apps, M. A. J., Argamon, S. E., …, Zwaan, R.A. (2018). “Justify your Alpha,” Nature Human Behavior, 2, 168–171. DOI: 10.1038/s41562-018-0311-x.
   Web of Science ®Google Scholar
• Leek, J.T. and Peng, R.D. (2015), “Statistics: p-Values are Just the Tip pf the Iceberg,” Nature, 520, 612, available at https://www.nature.com/polopoly_fs/1.17412!/menu/main/topColumns/topLeftColumn/pdf/520612a.pdf
   PubMed Web of Science ®Google Scholar
• Meehl, P.E. (1967), “Theory Testing in Psychology and in Physics: A Methodological Paradox,” Philosophy of Science, 34, 103–115. DOI: 10.1086/288135.
   Web of Science ®Google Scholar
• Nunnally, J. (1960), “The Place of Statistics in Psychology,” Educational and Psychological Measurement, 20, 641–650. DOI: 10.1177/001316446002000401.
   Web of Science ®Google Scholar
• Nuzzo, R. (2015), “Scientists Perturbed by Loss of Stat Tool to Sift Research Fudge from Fact,” Scientific American [online], available at https://www.scientificamerican.com/article/scientists-perturbed-by-loss-of-stat-toolstosift-research-fudge-from-fact/.
   Google Scholar
• Peng, R. (2015), “The Reproducibility Crisis in Science: A Statistical Counterattack,” Significance, 12(3), 30–32. DOI: 10.1111/j.1740-9713.2015.00827.x.
   Google Scholar
• Siegfried, T. (2015), “P Value Ban: Small Step for a Journal, Giant Leap for Science,” Science News [online], available at https://www.sciencenews.org/blog/context/p-value-ban-small-step-journal-giant-leapscience.
   Google Scholar
• Trafimow, D, Amrhein, V, Areshenkoff, C.N., Barrera-Causil, C.J., Beh, E.J., Bilgiç, Y.K., …, Marmolejo-Ramos, F. (2018) “Manipulating the Alpha Level Cannot Cure Significance Testing,” Frontiers in Psychology, 9, 699, available at https://www.frontiersin.org/articles/10.3389/fpsyg.2018.00699/full
   PubMed Web of Science ®Google Scholar
• Trafimow, D. and Marks, M. (2015), Editorial, Basic and Applied Social Psychology, 37(1), 1–2. DOI: 10.1080/01973533.2015.1012991.
   Web of Science ®Google Scholar
• Wasserstein, R.L., and Lazar, N.A. (2016a), “The ASA’s Statement on p-Values: Context, Process, and Purpose (Editorial, March 7, 2016),” The American Statistician, 70(2), 129–133.
   Web of Science ®Google Scholar
• Wasserstein, R.L., and Lazar, N.A. (2016b), “The ASA’s Statement on p-Values: Context, Process, and Purpose. Figshare, The American Statistician [online, supplemental materials] (March 7, 2016), available at https://figshare.com/collections/The_ASA’s_statement_on_p_values_context_process_and_purpose/2851090/1.
   Google Scholar
• Wetzels, R., Matzke, D., Lee, M.D., Rouder, J.N., Iverson G.J., and Wagenmakers, E.J. (2011), “Statistical Evidence in Experimental Psychology: An Empirical Comparison Using 855 t Tests,” Perspectives on Psychological Science, 6(3), 291–298. DOI: 10.1177/1745691611406923.
   Web of Science ®Google Scholar
• Woolston, C. (2015), “Psychology Journal Bans p-Values,” Nature, 519, 9, available at http://www.nature.com/news/psychologyjournalbans-p-values-1.17001.
   Google Scholar