Assessing the Religion-Health Relationship: Introduction to the Meta-analysis by Garssen et al., and Two Commentaries

Does religion promote well-being? And if so: which aspects of religiosity and well-being are involved? Religion can provide a sense of belonging, a sense of meaning and can instill self-regulation through its practices and rituals. Each of these factors could foster one’s mental and physical health. But religion can also instill fear and anxiety (Braam et al., 2000) Especially in more secular societies deviating from the cultural norm to be secular, being religious can actually be maladaptive for one’s health (Stavrova et al., 2013). The relationship between religion and health is multifaceted and this is further complicated by the fact that many studies in the field are cross-sectional, thereby obscuring the directionality of the relationship between religion and health.

In this issue of IJPR Garssen et al. (2020) report a rigorous meta-analysis of longitudinal studies assessing the relationship between religiosity and mental health. The rationale of this analysis is that cross-sectional studies can only provide limited evidence for causality. The main conclusion of the authors based on the meta-analytic results is that there is a positive but very small effect of religiosity on mental health: most notably religious participation and importance of religiosity are associated with reduced distress. Garssen et al., also observed a large heterogeneity in their outcomes, which might well be related to the diversity in the measures used in the different studies.

In response to the meta-analysis, two commentaries have been written which are published in this issue as well. VanderWeele (2020) commends the authors for their tremendous effort and reflects on the practical significance of the findings. VanderWeele argues that we have to face that the effect-sizes in the social sciences are small to moderate at best, but still at a population level small effect sizes can make a difference and can be meaningful. Koenig et al. (2020) provide a more critical assessment of the meta-analysis and disagrees with the conclusions of the authors based on both conceptual and methodological considerations. These concerns include, among others, contesting the choices for study-inclusion or exclusion, degrees of freedom in the design of the meta-analysis and the relevance of randomized controlled trials for establishing causality.

Garssen et al. (2020), in turn, have responded to these concerns. They point out that their meta-analysis follows similar guidelines and methods as used in previous meta-analyses by other research groups, including one by Braam and Koenig (2019). They further cast doubt on what counts as a medium or small effect-size by providing an illustrative example of how effect sizes translate into actual proportions. And they provide a crystal-clear justification for the design choices they made in setting up the meta-analysis. As such, this special issue provides an excellent example of a scholarly debate on an important topic, illustrating how even the experts in some cases can disagree.

At the same time, these responses and counter-responses dramatically illustrate that meta-analyses can never provide the evidence that will convince either the believer or the skeptic of the existence or absence of specific effects. Even though meta-analyses are still the gold-standard to aggregate findings in a field, the outcomes can be contested for many different reasons. For instance, in a previous publication we have shown that different meta-analytic techniques can yield different outcomes, providing either strong evidence for the absence or for the presence of a contested effect (Van Elk et al., 2015). Furthermore, Garssen et al. (2020) conclude that they did not find evidence for publication bias in the field of religion and mental health, but the methods they used for detecting bias, the fail-safe-N and the trim-and-fill procedure, are sub-optimal, especially in the case of large heterogeneity of effect sizes (Iyengar & Greenhouse, 1988; Peters et al., 2007). And although all authors in the current issue agree that it would be important to include only high-quality studies in a meta-analysis, as-of-yet we lack a system of checks-and-balances that allows in a straightforward way to assess the quality of each study that is included.

One way out of this battlefield is the use of open science practices, including pre-registration, registered reports, replication studies and many-analysts approaches. First, pre-registration offers a remedy against researcher degrees of freedom in the analysis and reporting of the results (Lakens, 2019). A recommendation for future meta-analyses on the religion-health relationship would be to include only studies that were pre-registered, as a simple and straightforward measure of quality control. Second, registered reports provide a direct way to counter publication-bias from the start, instead of controlling for this bias post-hoc. A registered report is a two-stage process in which the design of the study is reviewed prior to data collection, and if accepted, the paper will be published irrespective of the outcomes. Thirdly, large-scale independent replication studies offer the best alternative for meta-analyses to obtain evidence for the presence or absence of a contested effect (Van Elk et al., 2015). In that respect, it is illustrative that recent independent replication attempts of classical psychological studies have shown that the effect sizes obtained in replication studies were much smaller than the estimates from meta-analyses of these effects (Kvarven et al., 2020). As-of-yet we don’t know if the relationship between religion and health will fare any better, but the meta-analysis of Garssen et al., suggests that the effect size in longitudinal studies is small at best. Fourthly, given the large degrees of freedom in the way in which meta-analyses can be conducted, a many-analyst approach allows different researchers to conduct their analysis on the same data (Forscher et al., 2020). In this way it can be assessed to what extent outcomes, e.g., from meta-analyses, are robust to seemingly arbitrary choices in the design of the statistical tests (e.g., exclusion of outliers; using different methods to control for heterogeneity).

As an illustrative example of these open science practices, as part of the Religious Replication Project (RRP), we have recently conducted a large-scale preregistered cross-cultural replication study assessing the relationship between religion and well-being in 24 different countries and including more than 10.000 participants (Hoogeveen et al., in prep.). Instead of analyzing the outcomes ourselves, we have outsourced the analysis to more than 100 colleagues who each had to specify their specific analysis plan prior to getting access to the data. This way we tried to avoid bias in the analysis and reporting of results. Eventually, the outcomes of this project will shed new light on the boundary conditions under which religion could have a beneficial effect for one’s mental and physical health.

I would like to thank the authors who have contributed to this special issue for their efforts and accomplishments in this important research field. And it is my sincere hope that through the increased use of open science practices, ultimately, we will get a better understanding of the multifaceted relationship between being religious and feeling well.