The New Statistics for Better Science: Ask How Much, How Uncertain, and What Else Is Known

References

• American Psychological Association (2010), Publication Manual of the American Psychological Association, Washington, DC: Author.
   Google Scholar
• Benjamin, D. J., Berger, J. O., Johannesson, M., Nosek, B. A., Wagenmakers, E.-J., Berk, R., Bollen, K. A., Brembs, B., Brown, L., Camerer, C., Cesarini, D., Chambers, C. D., Clyde, M., Cook, T. D., De Boeck, P., Dienes, Z., Dreber, A., Easwaran, K., Efferson, C., Fehr, E., Fidler, F., Field, A. P., Forster, M., George, E. I., Gonzalez, R., Goodman, S., Green, E., Green, D. P., Greenwald, A. G., Hadfield, J. D., Hedges, L. V., Held, L., Hua Ho, T., Hoijtink, H., Hruschka, D. J., Imai, K., Imbens, G., Ioannidis, J. P. A., Jeon, M., Jones, J. H., Kirchler, M., Laibson, D., List, J., Little, R., Lupia, A., Machery, E., Maxwell, S. E., McCarthy, M., Moore, D. A., Morgan, S. L., Munafó, M., Nakagawa, S., Nyhan, B., Parker, T. H., Pericchi, L., Perugini, M., Rouder, J., Rousseau, J., Savalei, V., Schönbrodt, F. D., Sellke, T., Sinclair, B., Tingley, D., Van Zandt, T., Vazire, S., Watts, D. J., Winship, C., Wolpert, R. L., Xie, Y., Young, C., Zinman, J., and Johnson, V. E. (2018), “Redefine Statistical Significance,” Nature Human Behaviour, 2, 6–10. DOI: 10.1038/s41562-017-0189-z.
   PubMed Web of Science ®Google Scholar
• Bernard, C. (2017), “Editorial: A Message from the Editor-in-Chief,” Eneuro, 4, ENEURO.0023-17.2017. DOI: 10.1523/ENEURO.0023-17.2017.
   Web of Science ®Google Scholar
• Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., and Munafò, M. R. (2013), “Power Failure: Why Small Sample Size Undermines the Reliability of Neuroscience,” Nature Reviews Neuroscience, 14, 365–376. DOI: 10.1038/nrn3475.
   PubMed Web of Science ®Google Scholar
• Chavalarias, D., Wallach, J. D., Li, A. H. T., and Ioannidis, J. P. A. (2016), “Evolution of Reporting P Values in the Biomedical Literature, 1990–2015,” JAMA, 315, 1141. DOI: 10.1001/jama.2016.1952.
   PubMed Web of Science ®Google Scholar
• Cumming, G. (2011), Understanding the New Statistics: Effect Sizes, Confidence Intervals, and Meta-Analysis, New York: Routledge.
   Google Scholar
• Cumming, G., and Calin-Jageman, R. J. (2017), Introduction to the New Statistics: Estimation, Open Science, and Beyond, New York: Routledge.
   Google Scholar
• Cumming, G., and Finch, S. (2005), “Inference by Eye: Confidence Intervals and How to Read Pictures of Data,” The American Psychologist, 60, 170–180. DOI: 10.1037/0003-066X.60.2.170.
   PubMed Web of Science ®Google Scholar
• Davis, O. S. P., Band, G., Pirinen, M., Haworth, C. M. A., Meaburn, E. L., Kovas, Y., Harlaar, N., Docherty, S. J., Hanscombe, K. B., Trzaskowski, M., Curtis, C. J. C., Strange, A., Freeman, C., Bellenguez, C., Su, Z., Pearson, R., Vukcevic, D., Langford, C., Deloukas, P., Hunt, S., Gray, E., Dronov, S., Potter, S. C., Tashakkori-Ghanbaria, A., Edkins, S., Bumpstead, S. J., Blackwell, J. M., Bramon, E., Brown, M. A., Casas, J. P., Corvin, A., Duncanson, A., Jankowski, J. A. Z., Markus, H. S., Mathew, C. G., Palmer, C. N. A., Rautanen, A., Sawcer, S. J., Trembath, R. C., Viswanathan, A. C., Wood, N. W., Barroso, I., Peltonen, L., Dale, P. S., Petrill, S. A., Schalkwyk, L. S., Craig, I. W., Lewis, C. M., Price, T. S., Donnelly, P., Plomin, R., and Spencer, C. C. A. (2014), “The Correlation Between Reading and Mathematics Ability at Age Twelve Has a Substantial Genetic Component,” Nature Communications, 5, 4204.
   Google Scholar
• DeMayo, M. M., Song, Y. J. C., Hickie, I. B., and Guastella, A. J. (2017), “A Review of the Safety, Efficacy and Mechanisms of Delivery of Nasal Oxytocin in Children: Therapeutic Potential for Autism and Prader-Willi Syndrome, and Recommendations for Future Research,” Springer International Publishing, 19, 391–410. DOI: 10.1007/s40272-017-0248-y.
   Google Scholar
• Dumas-Mallet, E., Button, K. S., Boraud, T., Gonon, F., and Munafò, M. R. (2017), “Low Statistical Power in Biomedical Science: A Review of Three Human Research Domains,” Royal Society Open Science, 4, 160254. DOI: 10.1098/rsos.160254.
   PubMed Web of Science ®Google Scholar
• Eich, E. (2014), “Business Not as Usual,” Psychological Science, 25, 3–6. DOI: 10.1177/0956797613512465.
   PubMed Web of Science ®Google Scholar
• Fidler, F. (2010), “The American Psychological Association Publication Manual: Implications for Statistics Education (6th ed.),” Proceedings of ICOTS-8, Eighth International Conference on Teaching Statistics, 8.
   Google Scholar
• Fidler, F., Thomason, N., Cumming, G., Finch, S., and Leeman, J. (2004), “Editors Can Lead Researchers to Confidence Intervals, but Can’t Make Them Think: Statistical Reform Lessons from Medicine,” Psychological Science, 15, 119–126. DOI: 10.1111/j.0963-7214.2004.01502008.x.
   PubMed Web of Science ®Google Scholar
• Gilbert, D. T., King, G., Pettigrew, S., and Wilson, T. D. (2016), “Comment on ‘Estimating the Reproducibility of Psychological Science,”’ Science, 351, 1037–1037. DOI: 10.1126/science.aad7243.
   PubMed Web of Science ®Google Scholar
• Giofrè, D., Cumming, G., Fresc, L., Boedker, I., and Tressoldi, P. (2017), “The influence of Journal Submission Guidelines on Authors’ Reporting of Statistics and Use of Open Research Practices,” (J. M. Wicherts, ed.), PLOS ONE, 12, e0175583. DOI: 10.1371/journal.pone.0175583.
   PubMed Web of Science ®Google Scholar
• Greenland, S., Senn, S. J., Rothman, K. J., Carlin, J. B., Poole, C., Goodman, S. N., and Altman, D. G. (2016), “Statistical Tests, P-values, Confidence Intervals, and Power?: A Guide to Misinterpretations,” The American Statistician, 15, 1–31.
   Google Scholar
• Kelley, K. (2007), “Sample Size Planning for the Coefficient of Variation from the Accuracy in Parameter Estimation Approach,” Behavior Research Methods, 39, 755–766. DOI: 10.3758/BF03192966.
   PubMed Web of Science ®Google Scholar
• Kline, R. B. (2004), Beyond Significance Testing, Washington, DC: American Psychological Association.
   Google Scholar
• Kosfeld, M., Heinrichs, M., Zak, P. J., Fischbacher, U., and Fehr, E. (2005), “Oxytocin Increases Trust in Humans,” Nature, 435, 673–676. DOI: 10.1038/nature03701.
   PubMed Web of Science ®Google Scholar
• Kruschke, J. K. (2014), Doing Bayesian data analysis: A tutorial with R, JAGS, and Stan (2nd ed.).
   Google Scholar
• Kruschke, J. K., and Liddell, T. M. (2017), “The Bayesian New Statistics: Hypothesis Testing, Estimation, Meta-Analysis, and Power Analysis from a Bayesian Perspective,” Psychonomic Bulletin & Review.
   Google Scholar
• Lane, A., Luminet, O., Nave, G., and Mikolajczak, M. (2016), “Is there a Publication Bias in Behavioural Intranasal Oxytocin Research on Humans? Opening the File Drawer of One Laboratory,” Journal of Neuroendocrinology, 28.
   PubMed Web of Science ®Google Scholar
• Leng, G., and Ludwig, M. (2016a), “Review Intranasal Oxytocin?: Myths and Delusions,” Biological Psychiatry, 243–250. DOI: 10.1016/j.biopsych.2015.05.003.
   PubMed Web of Science ®Google Scholar
• Leng, G., and Ludwig, M. (2016b), “Reply to: Intranasal Oxytocin Mechanisms Can Be Better Understood, but Its Effects on Social Cognition and Behavior are Not to Be Sniffed At,” Biological Psychiatry, 79, e51–e52. DOI: 10.1016/j.biopsych.2015.06.022.
   PubMed Web of Science ®Google Scholar
• Makel, M. C., Plucker, J. A., and Hegarty, B. (2012), “Replications in Psychology Research: How Often Do They Really Occur?” Perspectives on Psychological Science, 7, 537–542. DOI: 10.1177/1745691612460688.
   Web of Science ®Google Scholar
• Maxwell, S. E., Kelley, K., and Rausch, J. R. (2008), “Sample Size Planning for Statistical Power and Accuracy in Parameter Estimation,” Annual Review of Psychology, 59, 537–563. DOI: 10.1146/annurev.psych.59.103006.093735.
   PubMed Web of Science ®Google Scholar
• McShane, B. B., and Gal, D. (2015), “Blinding Us to the Obvious? The Effect of Statistical Training on the Evaluation of Evidence,” Management Science, mnsc.2015.2212.
   PubMed Web of Science ®Google Scholar
• McShane, B. B., Gal, D., Gelman, A., Robert, C., and Tackett, J. L. (2017), “Abandon Statistical Significance,” arXiv preprint arXiv:1709.07588.
   Google Scholar
• Mellor, D. T., and Nosek, B. A. (2018), “Easy Preregistration will Benefit any Research,” Nature Human Behaviour, 2, 98–98. DOI: 10.1038/s41562-018-0294-7.
   Web of Science ®Google Scholar
• Nave, G., Camerer, C., and McCullough, M. (2015), “Does Oxytocin Increase Trust in Humans? A Critical Review of Research,” Perspectives on Psychological Science, 10, 772–789. DOI: 10.1177/1745691615600138.
   Web of Science ®Google Scholar
• Nelson, L. D., Simmons, J., and Simonsohn, U. (2018), “Psychology’s Renaissance,” Annual Review of Psychology, 511–534. DOI: 10.1146/annurev-psych-122216-011836.
   PubMed Web of Science ®Google Scholar
• Neumann, I. D., and Landgraf, R. (2012), “Balance of Brain Oxytocin and Vasopressin: Implications for Anxiety, Depression, and Social Behaviors,” Trends in Neurosciences, 35, 649–659. DOI: 10.1016/j.tins.2012.08.004.
   PubMed Web of Science ®Google Scholar
• Nieuwenhuis, S., Forstmann, B. U., and Wagenmakers, E.-J. (2011), “Erroneous Analyses of Interactions in Neuroscience: A Problem of Significance,” Nature Neuroscience, 14, 1105–1107. DOI: 10.1038/nn.2886.
   PubMed Web of Science ®Google Scholar
• Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S. J., Buck, S., Chambers, C. D., Chin, G., Christensen, G., Contestabile, M., Dafoe, A., Eich, E., Freese, J., Glennerster, R., Goroff, D., Green, D. P., Hesse, B., Humphreys, M., Ishiyama, J., Karlan, D., Kraut, A., Lupia, A., Mabry, P., Madon, T. A., Malhotra, N., Mayo-Wilson, E., McNutt, M., Miguel, E., Paluck, E. L., Simonsohn, U., Soderberg, C., Spellman, B. A., Turitto, J., VandenBos, G., Vazire, S., Wagenmakers, E. J., Wilson, R., and Yarkoni, T. (2015), “Promoting an Open Research Culture,” Science, 348, 1422–1425. DOI: 10.1126/science.aab2374.
   PubMed Web of Science ®Google Scholar
• Nosek, B. A., Ebersole, C. R., DeHaven, A. C., and Mellor, D. T. (2018), “The Preregistration Revolution,” Proceedings of the National Academy of Sciences, 115, 2600–2606. DOI: 10.1073/pnas.1708274114.
   PubMed Web of Science ®Google Scholar
• O’Boyle, E. H., Banks, G. C., and Gonzalez-Mule, E. (2014), “The Chrysalis Effect: How Ugly Initial Results Metamorphosize Into Beautiful Articles,” Journal of Management.
   Google Scholar
• Quintana, D. S. (2018), “Revisiting Non-Significant Effects of Intranasal Oxytocin Using Equivalence Testing,” Psychoneuroendocrinology, 87, 127–130. DOI: 10.1016/j.psyneuen.2017.10.010.
   PubMed Web of Science ®Google Scholar
• Quintana, D. S., and Woolley, J. D. (2016), “Intranasal Oxytocin Mechanisms Can Be Better Understood, but Its Effects on Social Cognition and Behavior Are Not to Be Sniffed At,” Biological Psychiatry, 79, e49–e50. DOI: 10.1016/j.biopsych.2015.06.021.
   PubMed Web of Science ®Google Scholar
• Shirani-Mehr, H., Rothschild, D., Goel, S., and Gelman, A. (2018), “Disentangling Bias and Variance in Election Polls,” Journal of the American Statistical Association, 113, 607–614 DOI: 10.1080/01621459.2018.1448823.
   Web of Science ®Google Scholar
• Smithson, M. (2002), Confidence Intervals, London: SAGE Publications.
   Google Scholar
• Sterling, T. D. (1959), “Publication Decisions and their Possible Effects on Inferences Drawn from Tests of Significance—or Vice Versa,” Journal of the American Statistical Association, 54, 30–34. DOI: 10.2307/2282137.
   Web of Science ®Google Scholar
• Sterling, T. D., Rosenbaum, W. L., and Weinkam, J. J. (1995), “Publication Decisions Revisited: The Effect of the Outcome of Statistical Tests on the Decision to Publish and Vice Versa,” The American Statistician, 49, 108. DOI: 10.2307/2684823.
   Web of Science ®Google Scholar
• Szucs, D., and Ioannidis, J. P. (2017), “Empirical Assessment of Published Effect Sizes and Power in The Recent Cognitive Neuroscience and Psychology Literature,” PLOS Biology, 15, e2000797. DOI: 10.1371/journal.pbio.2000797.
   PubMed Web of Science ®Google Scholar
• Tversky, A., and Kahneman, D. (1971), “Belief in the Law of Small Numbers,” Psychological Bulletin, 76, 105–110. DOI: 10.1037/h0031322.
   Google Scholar
• Walum, H., Waldman, I. D., and Young, L. J. (2016), “Statistical and Methodological Considerations for the Interpretation of Intranasal Oxytocin Studies,” Biological Psychiatry, 79, 251–257. DOI: 10.1016/j.biopsych.2015.06.016.
   PubMed Web of Science ®Google Scholar
• Woolley, J. D., Chuang, B., Lam, O., Lai, W., O’Donovan, A., Rankin, K. P., Mathalon, D. H., and Vinogradov, S. (2014), “Oxytocin Administration Enhances Controlled Social Cognition in Patients with Schizophrenia,” 47, 116–125. DOI: 10.1016/j.psyneuen.2014.04.024.
   Google Scholar
• Yatawara, C. J., Einfeld, S. L., Hickie, I. B., Davenport, T. A., and Guastella, A. J. (2016), “The Effect of Oxytocin Nasal Spray on Social Interaction Deficits Observed in Young Children with Autism: A Randomized Clinical Crossover Trial,” Molecular Psychiatry, 21, 1225–1231. DOI: 10.1038/mp.2015.162.
   PubMed Web of Science ®Google Scholar