In this special issue of Religion, Brain & Behavior we present the results of the Many-Analysts Religion Project (MARP). The MARP was conducted as part of the Religious Replication Project,Footnote1 a research initiative designed to critically evaluate the robustness of influential findings within the cognitive science of religion and the psychology of religion and spirituality. The Religious Replication Project sought to explore the extent of generalizability across various dimensions, including culture, religions, measurement techniques, and analytic traditions. From the outset, an important objective was not solely to promote the implementation of new tools and good research practices in the cognitive science of religion (Hoogeveen & van Elk, Citation2021), but also to put these recommendations into practice. This special issue provides a key example of that intention.
In the MARP, we delved into a longstanding inquiry within the psychology of religion: the relation between religiosity and well-being. For this project, we collected a large cross-cultural data set that included 10,535 participants from 24 countries and represented all major world religions. An important aspect of this project was our adoption of a many-analysts approach for which we invited research teams from various disciplinary backgrounds, ranging from religious studies and anthropology to methods and statistics, to independently analyze the data and address the key research questions.
The core idea of a many-analysts approach is to demonstrate the range of justifiable analytic decisions and their consequences in terms of outcomes and conclusions, thereby unveiling the robustness or fragility of the effect of interest. By addressing the same research question based on the same data, any differences in outcomes must be driven by differences in analytic decisions, where these decisions can be both theoretically or methodologically motivated. The many-analysts approach stands in opposition to two beguiling suggestions: (1) for any data set, there exists a single, uniquely appropriate analysis procedure and (2) multiple plausible analyses reliably yield similar conclusions (Wagenmakers et al., Citation2022). Previous many-analysts projects have already convincingly shown the mythical nature of these suggestions (Bastiaansen et al., Citation2020; Botvinik-Nezer et al., Citation2020; Silberzahn et al., Citation2018). In the current project, however, we wanted to do more than make a methodological point and provide a proof of concept for the many-analysts formula. Instead, we wanted to address a substantive question from the field, using the many-analysts approach as a valuable analysis tool, while also assessing analytic robustness. With this practical emphasis in mind, we chose to issue an open invitation to seek out analysts with both theoretical and methodological expertise to participate in the project. This resulted in a diverse pool of 120 analysis teams.
Among our cohort of analysts was Joseph Bulbulia who reached out to us while the project was still ongoing and kindly invited us to present our results within a dedicated special issue of Religion, Brain & Behavior. This format allowed us to display our main research findings, and also provided a framework for contextualizing these findings alongside the insights from various analysis teams. Publishing a main article, commentaries from participating analysis teams, and a reply afforded us ample opportunity to explain the intricacies of the challenges that arose during our project, explore the nuances of our findings, and present an in-depth analysis of the various aspects that underlie our results. Importantly, the format revealed aspects that might have remained hidden had we solely presented the outcomes of our primary analyses.
To the best of our knowledge, the present many-analysts effort is unique in that it provided participating analysis teams with extensive opportunity to share their perspectives. The added value of these comments has convinced us that it is important for a many-analysts effort to include into the results and discussion sections both the teams’ quantitative results as well as their qualitative insights and critiques.
The first article in this special issue, Hoogeveen, Sarafoglou, Aczel, et al. (Citation2022), contains the introduction of the MARP as well as the main outcomes for our two research questions (1) “Do religious people self-report greater well-being?” and (2) “Does the relation between religiosity and self-reported well-being depend on perceived cultural norms of religion?”. To anticipate our main results, for the first research question, we found that all but three teams reported positive effect sizes with credible/confidence intervals excluding zero. As for the second research question, while the outcomes were somewhat less consistent, they still suggest support in favor for the hypothesis: 95% of the teams reported a positive effect size for the moderating role of cultural norms of religion on the association between religiosity and well-being, with 65% of the credible/confidence intervals excluding zero.
Following the main article are a series of commentaries from participating analysis teams. These commentaries offer nuanced interpretations of the results, present findings from additional tested hypotheses, and discuss the robustness and generalizability of the data set. Some commentaries critically analyze the methodology and set-up of the many-analysts approach, while others highlight concerns about the data set itself. In line with the focus of Religion, Brain & Behavior, as well as the aim of the MARP, the commentaries address both substantive and methodological angles of the project. The special issue concludes with a discussion and conclusion (Hoogeveen, Sarafoglou, van Elk, et al., Citation2022), in which we respond to the commentaries of the analysis teams and reflect on possible future directions in this field of research, especially with regard to possible methodological refinements.
What can we conclude regarding the relation between religiosity and well-being? While we believe the analytic robustness and consistency of the results is remarkable—especially for the overall relation between religion and self-reported well-being– the discussion on the relation between religiosity and well-being is by no means settled.
Indeed, two important limitations of the MARP findings are that they relied on self-report measures and non-experimental cross-sectional data. As such, the question of how to interpret the identified associations persists; as the adage goes, correlation does not imply causation. Does religion truly contribute to people’s happiness? Does attending religious services alleviate anxiety, or does anxiety hinder participation in religious gatherings (Bulbulia, Citation2022)? Or are third variables such as socio-economic status and upbringing driving the associations (Atkinson et al., Citation2022; van Lissa, Citation2022)? Longitudinal studies, quasi-experimental designs (e.g., comparing students in religious vs. public schools) or other methods for causal inference (Bulbulia, Citation2022; Pearl, Citation1995; Rohrer, Citation2018) appear vital in moving this discussion forward. In addition, the issue of measurement invariance across cultures is arguably crucial for valid inferences yet often ignored (Fischer & Karl, Citation2019; Hussey & Hughes, Citation2020; Ross et al., Citation2022; Schreiner et al., Citation2022). We believe it is important for future cross-cultural research on religion and well-being to validate that measures of religiosity or mental health, for instance in Japan, are understood in a comparable manner to those in the US. Practical tools to assess multi-group invariance (e.g., Fischer & Karl, Citation2019) may be instrumental in this development.
In terms of methodological take-home messages, we believe it is important to acknowledge that despite the qualitatively similar results obtained by the various teams in the MARP, the variability across teams is still considerably larger than the variability within teams. This suggests that individual estimates are typically over-confident, even when taking uncertainty into account, and emphasizes again the merit of reporting multiple analyses for a given question and data set.
Based on our personal experiences in orchestrating a many-analysts project, we advise team leaders to pay special attention to how they synthesize the evidence in their many-analysts project. The engagement of the analysis teams within the MARP and their motivation to explore the research questions from various angles showed us that it is worthwhile to include the qualitative insights of the analysis teams alongside the reported effect sizes in upcoming projects of this nature. To achieve this, a feasible strategy could involve administering a survey to analysis teams that solicits their methodological concerns and subjective assessments of the evidence for the research questions. An additional area for future research would be to explore strategies for efficiently summarizing analysis teams’ quantitative results. Such an approach should aim to generalize across various effect sizes while also accounting for dependencies among analytic outcomes that stem from the same data set, in a kind of internal meta-analysis.
These issues and open questions notwithstanding, we believe the many-countries, many-religions, many-analysts approach in the MARP provides a compelling illustration that the self-reported, non-causal relationship between religion and well-being seems robust, especially for mental well-being. We hope the MARP may serve as an example and inspire future team-science projects.
Notes
References
- Atkinson, Q. D., Claessens, S., Fischer, K., Forsyth, G. L., Kyritsis, T., Wiebels, K., & Moreau, D. (2022). Being specific about generalisability. Religion, Brain & Behavior, 284–286. https://doi.org/10.1080/2153599X.2022.2070251
- Bastiaansen, J. A., Kunkels, Y. K., Blaauw, F. J., Boker, S. M., Ceulemans, E., Chen, M., Chow, S.-M., de Jonge, P., Emerencia, A. C., Epskamp, S., Fisher, A. J., Hamaker, E. L., Kuppens, P., Lutz, W., Meyer, M. J., Moulder, R., Oravecz, Z., Riese, H., Rubel, J., … Bringmann, L. F. (2020). Time to get personal? The impact of researchers choices on the selection of treatment targets using the experience sampling methodology. Journal of Psychosomatic Research, 137, 110211. https://doi.org/10.1016/j.jpsychores.2020.110211
- Botvinik-Nezer, R., Holzmeister, F., Camerer, C. F., Dreber, A., Huber, J., Johannesson, M., Kirchler, M., Iwanir, R., Mumford, J. A., Adcock, R. A., Avesani, P., Baczkowski, B. M., Bajracharya, A., Bakst, L., Ball, S., Barilari, M., Bault, N., Beaton, D., Beitner, J., … Schonberg, T. (2020). Variability in the analysis of a single neuroimaging dataset by many teams. Nature, 582(7810), 84–88. https://doi.org/10.1038/s41586-020-2314-9
- Bulbulia, J. A. (2022). A workflow for causal inference in cross-cultural psychology. Religion, Brain & Behavior, 291–306. https://doi.org/10.1080/2153599X.2022.2070245
- Fischer, R., & Karl, J. A. (2019). A primer to (cross-cultural) multi-group invariance testing possibilities in R. Frontiers in Psychology, 10. https://doi.org/10.3389/fpsyg.2019.01507
- Hoogeveen, S., Sarafoglou, A., Aczel, B., Aditya, Y., Alayan, A., Allen, P., Altay, S., Alzahawi, S., Amir, Y., Anthony, F.-V., Appiah, O., Atkinson, Q. D., Baimel, A., Balkaya-Ince, M., Balsamo, M., Banker, S., Bartoš, F., Becerra, M., Beffara, B., … Wagenmakers, E.-J. (2022). A many-analysts approach to the relation between religiosity and well-being. Religion, Brain & Behavior, 237–283. https://doi.org/10.31234/osf.io/pbfye
- Hoogeveen, S., Sarafoglou, A., van Elk, M., & Wagenmakers, E.-J. (2022). Many-Analysts Religion Project: Reflection and conclusion. Religion, Brain & Behavior, 356–363.
- Hoogeveen, S., & van Elk, M. (2021). Advancing the cognitive science of religion through replication and open science. Journal for the Cognitive Science of Religion, 6, 158–190. https://doi.org/10.1558/jcsr.39039
- Hussey, I., & Hughes, S. (2020). Hidden invalidity among 15 commonly used measures in social and personality psychology. Advances in Methods and Practices in Psychological Science, 3(2), 166–184. https://doi.org/10.1177/2515245919882903
- Pearl, J. (1995). Causal diagrams for empirical research. Biometrika, 82(4), 669–688. doi:10.1093/biomet/82.4.669
- Rohrer, J. M. (2018). Thinking clearly about correlations and causation: Graphical causal models for observational data. Advances in Methods and Practices in Psychological Science, 1(1), 27–42. https://doi.org/10.1177/2515245917745629
- Ross, R. M., Sulik, J., Buczny, J., & Schivinski, B. (2022). Many analysts and few incentives. Religion, Brain & Behavior, 1–3. https://doi.org/10.1080/2153599X.2022.2070248
- Schreiner, M. R., Mercier, B., Frick, S., Wiwad, D., Schmitt, M. C., Kelly, J. M., & Quevedo Pütter, J. (2022). Measurement issues in the Many Analysts Religion Project. Religion, Brain & Behavior, 339–341. https://doi.org/10.1080/2153599X.2022.2070260
- Silberzahn, R., Uhlmann, E. L., Martin, D. P., Anselmi, P., Aust, F., Awtrey, E., Bahnik, Š, Bai, F., Bannard, C., Bonnier, E., Carlsson, R., Cheung, F., Christensen, G., Clay, R., Craig, M. A., Dalla Rosa, A., Dam, L., Evans, M. H., Flores Cervantes, I., … Nosek, B. A. (2018). Many analysts, one data set Making transparent how variations in analytic choices affect results. Advances in Methods and Practices in Psychological Science, 1(3), 337–356. https://doi.org/10.1177/2515245917747646
- van Lissa, C. J. (2022). Complementing preregistered confirmatory analyses with rigorous, reproducible exploration using machine learning. Religion, Brain & Behavior, 347–351. doi.org/10.1080/2153599X.2022.2070254
- Wagenmakers, E.-J., Sarafoglou, A., & Aczel, B. (2022). One statistical analysis must not rule them all. Nature, 605(7910), 423–425. https://doi.org/10.1038/d41586-022-01332-8