Researcher Degrees of Freedom in the Psychology of Religion

References

• Allen, C. P. G., & Mehler, D. M. (2018). Open Science challenges, benefits and tips in early career and beyond ( Unpublished manuscript). doi:10.31234/osf.io/3czyt
   Google Scholar
• Ambasciano, L., & Coleman, T. J., III. (2019). History as a canceled problem? Hilbert lists, du Bois-Reymond’s enigmas, and the scientific study of religion. Journal of the American Academy of Religion, 87, 366–400. doi:10.1093/jaarel/lfz001
   Web of Science ®Google Scholar
• Asarnow, J., Bloch, M. H., Brandeis, D., Burt, S. A., Fearon, P., Fombonne, E., … Zeanah, C. H. (2018). Special editorial: Open science and the journal of child psychology & psychiatry – next steps? Journal of Child Psychology and Psychiatry, 59(7), 826–827. doi:10.1111/jcpp.12929
   PubMed Web of Science ®Google Scholar
• Bains, S. (2011). Questioning the integrity of the John Templeton Foundation. Evolutionary Psychology, 9(1), 147470491100900111. doi:10.1177/147470491100900111
   Web of Science ®Google Scholar
• Bajpai, S., Bajpai, R. C., & Chaturvedi, H. K. (2015). Evaluation of inter-rater agreement and inter-rater reliability for observational data: An overview of concepts and methods’. Journal of the Indian Academy of Applied Psychology, 41(3), 20.
   Google Scholar
• Bakker, M., Hartgerink, C. H. J., Wicherts, J. M., & van der Maas, H. L. J. (2016). Researchers’ intuitions about power in psychological research. Psychological Science, 27(8), 1069–1077. doi:10.1177/0956797616647519
   PubMed Web of Science ®Google Scholar
• Bakker, M., van Dijk, A., & Wicherts, J. M. (2012). The rules of the game called psychological science. Perspectives on Psychological Science, 7(6), 543–554. doi:10.1177/1745691612459060
   Web of Science ®Google Scholar
• Bender, R., & Lange, S. (2001). Adjusting for multiple testing - When and how? Journal of Clinical Epidemiology, 54(4), 343–349. doi:10.1016/S0895-4356(00)00314-0
   PubMed Web of Science ®Google Scholar
• Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society. Series B (methodological), 57, 289–300. doi:10.1111/rssb.1995.57.issue-1
   Web of Science ®Google Scholar
• Bland, J. M., & Altman, D. G. (1995). Multiple significance tests: The Bonferroni method. BMJ, 310(6973), 170. doi:10.1136/bmj.310.6973.170
   PubMed Web of Science ®Google Scholar
• Brown, N. J. L., & Heathers, J. A. J. (2017). The GRIM test: A simple technique detects numerous anomalies in the reporting of results in psychology. Social Psychological and Personality Science, 8(4), 363–369. doi:10.1177/1948550616673876
   Web of Science ®Google Scholar
• Button, K. S., Ioannidis, J. P. A., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S. J., & Munafò, M. R. (2013). Power failure: Why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience, 14, 365–376. doi:10.1038/nrn3475
   PubMed Web of Science ®Google Scholar
• Camerer, C. F., Dreber, A., Holzmeister, F., Ho, T.-H., Huber, J., Johannesson, M., … Chambers, C. D. (2013). Registered reports: A new publishing initiative at Cortex. Cortex, 49(3), 609–610. doi:10.1016/j.cortex.2012.12.016
   PubMed Web of Science ®Google Scholar
• Chambers, C. D., Feredoes, E., Muthukumaraswamy, S. D., & Etchells, P. (2014). Instead of “playing the game” it is time to change the rules: Registered reports at AIMS neuroscience and beyond. AIMS Neuroscience, 1, 4–17. doi:10.3934/Neuroscience.2014.1.4
   Google Scholar
• Champely, S., Ekstrom, C., Dalgaard, P., Gill, J., Weibelzahl, S., Anandkumar, A., … De Rosario, M. H. (2018). R package version, 1-2.
   Google Scholar
• Cohen, J. (1962). The statistical power of abnormal-social psychological research: a review. The Journal of Abnormal and Social Psychology, 65(3), 145. doi:10.1037/h0045186
   PubMed Web of Science ®Google Scholar
• Coleman, T. J., III, & Jong, J. (in press). Counting the nonreligious. In A. L. Ai, K. A. Harris, & P. Wink (Eds.), Assessing spirituality and religion in a diversified world: Beyond the mainstream perspective. Cham, Switzerland: Springer.
   Google Scholar
• Coleman, T. J., III, & Hood, R. W., Jr. (2015). Reconsidering everything: From folk categories to existential theory of mind. [Peer commentary on the paper “from weird experiences to revelatory events” by A. Taves]. Religion and Society: Advances in Research, 6(1), 18–22.
   Google Scholar
• Cramer, A. O. J., van Ravenzwaaij, D., Matzke, D., Steingroever, H., Wetzels, R., Grasman, R. P. P. P., & Wagenmakers, E. J. (2016). Hidden multiplicity in exploratory multiway ANOVA: Prevalence and remedies. Psychonomic Bulletin & Review, 23(2), 640–647. doi:10.3758/s13423-015-0913-5
   PubMed Web of Science ®Google Scholar
• de Groot, A. D. (1954/2014). The meaning of “significance” for different types of research [translated and annotated by Eric-Jan Wagenmakers, Denny Borsboom, Josine Verhagen, Rogier Kievit, Marjan Bakker, Angelique Cramer, Dora Matzke, Don Mellenbergh, and Han L. J. van Der Maas]. Acta Psychologica, 148, 188–194. doi:10.1016/j.actpsy.2014.02.001
   Google Scholar
• de Jager Meezenbroek, E., Garssen, B., van Den Berg, M., Van Dierendonck, D., Visser, A., & Schaufeli, W. B. (2012). Measuring spirituality as a universal human experience: A review of spirituality questionnaires. Journal of Religion and Health, 51(2), 336–354. doi:10.1007/s10943-010-9376-1
   PubMed Web of Science ®Google Scholar
• Etz, A., & Vandekerckhove, J. (2016). A Bayesian perspective on the reproducibility project: Psychology. PLoS ONE, 11(2), 1–12. doi:10.1371/journal.pone.0149794
   Web of Science ®Google Scholar
• Fanelli, D. (2010). “Positive” results increase down the hierarchy of the sciences. PLoS ONE, 5(4), 1–10. doi:10.1371/journal.pone.0010068
   Google Scholar
• Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39, 175–191.
   PubMed Web of Science ®Google Scholar
• Fernández, L. M., & Vadillo, M. A. (2019). Reaction times: Many ways of inadvertently obtaining a false positive. doi:10.31219/osf.io/d4yqz
   Google Scholar
• Flake, J. K., Pek, J., & Hehman, E. (2017). Construct validation in social and personality research: Current practice and recommendations. Social Psychological and Personality Science, 8(4), 370–378. doi:10.1177/1948550617693063
   Web of Science ®Google Scholar
• Gelman, A., & Loken, E. (2013). The garden of forking paths: Why multiple comparisons can be a problem, even when there is no “fishing expedition” or “p-hacking” and the research hypothesis was posited ahead of time. USA: Department of Statistics, Columbia University.
   Google Scholar
• Goldacre, B., Drysdale, H., Dale, A., Milosevic, I., Slade, E., Hartley, P., … Mahtani, K. R. (2019). COMPare: A prospective cohort study correcting and monitoring 58 misreported trials in real time. Trials, 20(1), 118. doi:10.1186/s13063-019-3242-6
   PubMed Web of Science ®Google Scholar
• Guo, Q., Thabane, L., Hall, G., McKinnon, M., Goeree, R., & Pullenayegum, E. (2014). A systematic review of the reporting of sample size calculations and corresponding data components in observational functional magnetic resonance imaging studies. NeuroImage, 86, 172–181. doi:10.1016/j.neuroimage.2013.08.012
   PubMed Web of Science ®Google Scholar
• Gwet, K. L. (2008). Computing inter-rater reliability and its variance in the presence of high agreement. British Journal of Mathematical and Statistical Psychology, 61(1), 29–48. doi:10.1348/000711006X126600
   PubMed Web of Science ®Google Scholar
• Head, M. L., Holman, L., Lanfear, R., Kahn, A. T., & Jennions, M. D. (2015). The extent and consequences of p-hacking in science. PLoS Biology, 13(3), e1002106. doi:10.1371/journal.pbio.1002106
   PubMed Web of Science ®Google Scholar
• HyLown Consulting LLC. (2018). Power and sample size. Retrieved from http://www.powerandsamplesize.com
   Google Scholar
• John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23(5), 524–532. doi:10.1177/0956797611430953
   PubMed Web of Science ®Google Scholar
• Kappenman, E. S., & Keil, A. (2017). Introduction to the special issue on recentering science: Replication, robustness, and reproducibility in psychophysiology. Psychophysiology, 54(1), 3–5. doi:10.1111/psyp.12787
   PubMed Web of Science ®Google Scholar
• Kidwell, M. C., Lazarević, L. B., Baranski, E., Hardwicke, T. E., Piechowski, S., Falkenberg, L.-S., … Nosek, B. A. (2016). Badges to acknowledge open practices: A simple, low-cost, effective method for increasing transparency. PLoS Biology, 14(5), e1002456. doi:10.1371/journal.pbio.1002456
   PubMed Web of Science ®Google Scholar
• Klein, O., Hardwicke, T. E., Aust, F., Breuer, J., Danielsson, H., Mohr, A. H., … Frank, M. C. (2018). A practical guide for transparency in psychological science. Collabra: Psychology, 4(1), 20. doi:10.1525/collabra.158
   Google Scholar
• Lachlan, K., & Spence, P. R. (2005). Corrections for type I error in social science research: A disconnect between theory and practice. Journal of Modern Applied Statistical Methods, 5(2), 490–494. doi:10.22237/jmasm/1162354920
   Google Scholar
• Ladd, K. L., & Messick, K. J. (2016). A brief history of the psychological study of the role(s) of religion. In W. Woody, R. Miller, & W. Wozniak (Eds.), Psychological specialties in historical context: Enriching the classroom experience for teachers and students (pp. 204–216). Division 2, American Psychological Association. Retrieved from https://teachpsych.org/ebooks/psychspec
   Google Scholar
• Ladd, K. L. (2019). The archive for the psychology of religion: Editorial principles, practices, and practicalities. Archive for the Psychology of Religion, 41(1), 3–11. doi:10.1177/0084672419828844
   Google Scholar
• Lakens, D. (2014). Performing high-powered studies efficiently with sequential analyses. European Journal of Social Psychology, 44(7), 701–710. doi:10.1002/ejsp.2023
   Web of Science ®Google Scholar
• Lane, A., Luminet, O., Nave, G., & Mikolajczak, M. (2016). Is there a publication bias in behavioural intranasal oxytocin research on humans? Opening the file drawer of one laboratory. Journal of Neuroendocrinology, 28(4), 1–15. doi:10.1111/jne.12384
   Web of Science ®Google Scholar
• Larson, M. J., & Carbine, K. A. (2017). Sample size calculations in human electrophysiology (EEG and ERP) studies: A systematic review and recommendations for increased rigor. International Journal of Psychophysiology, 111, 33–41. doi:10.1016/j.ijpsycho.2016.06.015
   PubMed Web of Science ®Google Scholar
• Latour, B., & Woolgar, S. (1979/2013). Laboratory life: The construction of scientific facts (2nd ed.). Princeton, NJ: Princeton University Press.
   Google Scholar
• Lo Martire, R. (2017). Rel: Reliability coefficients. R package version 1.3.1. Retrieved from https://CRAN.R-project.org/package=rel
   Google Scholar
• Messick, K., & Farias, M. (in press). The psychology of leaving religion. In Brill handbook of leaving religion. Brill.
   Google Scholar
• Miguel, E., Camerer, C., Casey, K., Cohen, J., Esterling, K. M., Gerber, A., … Laitin, D. (2014). Promoting transparency in social science research. Science, 343(6166), 30–31. doi:10.1126/science.1245317
   PubMed Web of Science ®Google Scholar
• Moberg, D. O. (2002). Assessing and measuring spirituality: Confronting dilemmas of universal and particular evaluative criteria. Journal of Adult Development, 9(1), 47–60. doi:10.1023/A:1013877201375
   Web of Science ®Google Scholar
• Morey, R. D., & Lakens, D. (2016). Why most of psychology is statistically unfalsifiable. Retrieved from https://github.com/richarddmorey/psychology_resolution/blob/master/paper/response.pdf
   Google Scholar
• Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie Du Sert, N., … Ioannidis, J. P. A. (2017). A manifesto for reproducible science. Nature Human Behaviour, 1(1), 0021. doi:10.1038/s41562-016-0021
   PubMed Web of Science ®Google Scholar
• Nelson, L. D., Simmons, J., & Simonsohn, U. (2018). Psychology’s renaissance. Annual Review of Psychology, 69(1), 511–534. doi:10.1146/annurev-psych-122216-011836
   PubMed Web of Science ®Google Scholar
• Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences, 115, 2600–2606.
   PubMed Web of Science ®Google Scholar
• Nosek, B. A., & Lakens, D. (2014). Registered reports. Social Psychology, 45(3), 137–141. doi:10.1027/1864-9335/a000192
   Web of Science ®Google Scholar
• Nosek, B. A., & Lindsay, D. S. (2018). Preregistration becoming the norm in psychological science. APS Observer, 31(30/3). Retrieved from https://www.psychologicalscience.org/observer/preregistration-becoming-the-norm-in-psychological-science.
   Google Scholar
• Nuijten, M. B. (2019). Practical tools and strategies for researchers to increase replicability. Developmental Medicine & Child Neurology, 61(5), 535–539. doi:10.1111/dmcn.14054
   PubMed Web of Science ®Google Scholar
• Nuijten, M. B., Hartgerink, C. H. J., van Assen, M. A. L. M., Epskamp, S., & Wicherts, J. M. (2016). The prevalence of statistical reporting errors in psychology (1985–2013). Behavior Research Methods, 48(4), 1205–1226. doi:10.3758/s13428-014-0555-y
   PubMed Web of Science ®Google Scholar
• Olsson-Collentine, A., van Assen, M. A., & Hartgerink, C. H. (2019). The prevalence of marginally significant results in psychology over time. Psychological Science, 30, 576–586.
   PubMed Web of Science ®Google Scholar
• Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), 1–8. doi:10.1126/science.aac4716
   Web of Science ®Google Scholar
• R Core Team. (2017). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. [Computer software]. Retrieved from https://www.R-project.org/
   Google Scholar
• Roettger, T. B. (2018). Researcher degrees of freedom in phonetic sciences. Retrieved from www.psyarxiv.com/fp4jr
   Google Scholar
• Rouse, S. V. (2018). Introduction to the special issue on open science practices: Badges of honor. Psi Chi Journal of Psychological Research, 23(2), 94–97. doi:10.24839/2325-7342.JN
   Google Scholar
• Rowhani-Farid, A., Allen, M., & Barnett, A. G. (2017). What incentives increase data sharing in health and medical research? A systematic review. Research Integrity and Peer Review, 2(1), 4. doi:10.1186/s41073-017-0028-9
   PubMedGoogle Scholar
• Schäfer, T., & Schwarz, M. A. (2019). The meaningfulness of effect sizes in psychological research: Differences between sub-disciplines and the impact of potential biases. Frontiers in Psychology, 10. doi:10.3389/fpsyg.2019.00813
   Web of Science ®Google Scholar
• Sedlmeier, P., & Gigerenzer, G. (1989). Do studies of statistical power have an effect on the power of studies?. Psychological Bulletin, 105(2), 309–316. doi:10.1037/0033-2909.105.2.309
   Web of Science ®Google Scholar
• Silberzahn, R., Uhlmann, E. L., Martin, D. P., Anselmi, P., Aust, F., Awtrey, E., & Carlsson, R. (2018). Many analysts, one data set: making transparent how variations in analytic choices affect results. Advances in Methods and Practices in Psychological Science, 1(3), 337–356.
   Google Scholar
• Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359–1366. doi:10.1177/0956797611417632
   PubMed Web of Science ®Google Scholar
• Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2017, November 6). How to properly preregister a study [blog post]. Retrieved from http://datacolada.org/64
   Google Scholar
• Skinner, I. W., Hübscher, M., Moseley, G. L., Lee, H., Wand, B. M., Traeger, A. C., … McAuley, J. H. (2017). The reliability of eyetracking to assess attentional bias to threatening words in healthy individuals. Behavior Research Methods, 50(5), 1778–1792. doi:10.3758/s13428-017-0946-y
   Web of Science ®Google Scholar
• Slater, W., Hall, T. W., & Edwards, K. J. (2001). Measuring religion and spirituality: Where are we and where are we going? Journal of Psychology and Theology, 29(1), 4–21. doi:10.1177/009164710102900102
   Web of Science ®Google Scholar
• Tamminen, K. A., & Poucher, Z. A. (2018). Open science in sport and exercise psychology: Review of current approaches and considerations for qualitative inquiry. Psychology of Sport and Exercise, 36, 17–28. doi:10.1016/j.psychsport.2017.12.010
   Web of Science ®Google Scholar
• Turner, L., Shamseer, L., Altman, D. G., Schulz, K. F., & Moher, D. (2012). Does use of the CONSORT Statement impact the completeness of reporting of randomised controlled trials published in medical journals? A Cochrane review. Systematic Reviews, 1(1), 60. doi:10.1186/2046-4053-1-60
   PubMedGoogle Scholar
• Uzdavines, A., Hill, P., Coleman, T. J., III, Gibson, N. J., & Stauner, N. (2017). Open science ideals, practices, and dissemination within the international psychology of religion community. Presentation conducted at the 2017 International Association for the Psychology of Religion World Congress, Hamar, Norway.
   Google Scholar
• van Elk, M., Rowatt, W., & Streib, H. (2018). Good dog, bad dog: Introducing open science badges. The International Journal for the Psychology of Religion, 28(1), 1–2. doi:10.1080/10508619.2018.1402589
   Web of Science ®Google Scholar
• van’t Veer, A. E., & Giner-Sorolla, R. (2016). Pre-registration in social psychology—A discussion and suggested template. Journal of Experimental Social Psychology, 67, 2–12. doi:10.1016/j.jesp.2016.03.004
   Web of Science ®Google Scholar
• Viergever, R. F., & Li, K. (2015). Trends in global clinical trial registration: An analysis of numbers of registered clinical trials in different parts of the world from 2004 to 2013. BMJ Open, 5(9), e008932. doi:10.1136/bmjopen-2015-008932
   PubMed Web of Science ®Google Scholar
• Wagenmakers, E. J., Beek, T., Dijkhoff, L., Gronau, Q. F., Acosta, A., Adams Jr, R. B., … Bulnes, L. C. (2016). Registered replication report: Strack, Martin, & Stepper (1988). Perspectives on Psychological Science, 11(6), 917–928. doi:10.1177/1745691616674458
   Web of Science ®Google Scholar
• Wagenmakers, E. J., Wetzels, R., Borsboom, D., van der Maas, H. L., & Kievit, R. A. (2012). An agenda for purely confirmatory research. Perspectives on Psychological Science, 7(6), 632–638. doi:10.1177/1745691612463078
   Web of Science ®Google Scholar
• Wicherts, J. M., Veldkamp, C. L. S., Augusteijn, H. E. M., Bakker, M., van Aert, R. C. M., & van Assen, M. A. L. M. (2016). Degrees of freedom in planning, running, analyzing, and reporting psychological studies: A checklist to avoid p-hacking. Frontiers in Psychology, 7, 1–12. doi:10.3389/fpsyg.2016.00001
   PubMed Web of Science ®Google Scholar
• Wongpakaran, N., Wongpakaran, T., Wedding, D., & Gwet, K. L. (2013). A comparison of Cohen’s Kappa and Gwet’s AC1 when calculating inter-rater reliability coefficients: A study conducted with personality disorder samples. BMC Medical Research Methodology, 13(61), 1–7. doi:10.1186/1471-2288-13-1
   PubMedGoogle Scholar
• Wulff, D. (1998). Rethinking the rise and fall of the psychology of religion. In A. Molendijk & P. Peels (Eds.), Religion in the making: The emergence of the sciences of religion (pp. 181–202). Leiden, Netherlands: Brill.
   Google Scholar
• Wulff, D. (2003). A field in crisis: Is it time for the psychology of religion to start over? In P. Roelofsma, J. Corveleyn, & J. Van Saane (Eds.), One hundred years of psychology and religion: Issues and trends in a century long quest (pp. 11–32). Amsterdam, The Netherlands: VU University Press.
   Google Scholar