A many-analysts approach to the relation between religiosity and well-being

ABSTRACT

The relation between religiosity and well-being is one of the most researched topics in the psychology of religion, yet the directionality and robustness of the effect remains debated. Here, we adopted a many-analysts approach to assess the robustness of this relation based on a new cross-cultural dataset ($N=10,535$ participants from 24 countries). We recruited 120 analysis teams to investigate (1) whether religious people self-report higher well-being, and (2) whether the relation between religiosity and self-reported well-being depends on perceived cultural norms of religion (i.e., whether it is considered normal and desirable to be religious in a given country). In a two-stage procedure, the teams first created an analysis plan and then executed their planned analysis on the data. For the first research question, all but 3 teams reported positive effect sizes with credible/confidence intervals excluding zero (median reported $\beta =0.120$). For the second research question, this was the case for 65% of the teams (median reported $\beta =0.039$). While most teams applied (multilevel) linear regression models, there was considerable variability in the choice of items used to construct the independent variables, the dependent variable, and the included covariates.

KEYWORDS:

• Health
• many analysts
• open science
• religion

1. Introduction

The relation between religion and well-being has been a topic of debate for centuries. While Freud considered religion a “universal obsessional neurosis” and Nietzsche called Christianity “the greatest misfortune of humanity,” the recent scientific literature has painted a more positive picture of religion's effect on (mental) health (e.g., Gebauer et al., 2017; George et al., 2002; Koenig & Larson, 2001; Plante & Sherman, 2001; Seybold & Hill, 2001; Thoresen, 1999; Zimmer et al., 2016). Individual religiosity has, for instance, been related to less depression (Smith et al., 2003), more happiness (Abdel-Khalek, 2006; Lewis & Cruise, 2006), higher life satisfaction (Lim & Putnam, 2010), and even lower mortality (Ebert et al., 2020; Stavrova, 2015). At the same time, the robustness, universality, and methodological specificity of the religion—well-being relation remains an outstanding question. In this project, we adopted a many-analysts approach to investigate two research questions using a new large cross-cultural dataset featuring $N=10,535$ participants from 24 countries. Specifically, we recruited 120 teams to conduct analyzes in order to answer the following two research questions: (1) “Do religious people self-report greater well-being?,” and (2) “Does the relation between religiosity and self-reported well-being depend on perceived cultural norms regarding religion?.” In the subsequent sections, we will first introduce our theoretical framework, dataset, and the many-analysts approach, before describing the key results with respect to the stated research questions and the varying approaches taken by the many-analysts teams. A general discussion of the project and the results is included in the closing article (Hoogeveen et al., 2022).

1.1. Theoretical background

The literature on the psychology of religion is replete with positive correlations between (self-rated) religiosity and mental health (Abdel-Khalek, 2006; George et al., 2002; Koenig & Larson, 2001; Plante & Sherman, 2001; Seybold & Hill, 2001; Smith et al., 2003; Thoresen, 1999; Zimmer et al., 2016; see Koenig, 2009 for a review). At the same time, meta-analyzes indicate that the relation between religion and well-being is often small (around r = .1; Bergin, 1983; Hackney & Sanders, 2003; Koenig & Larson, 2001). In addition, it has been argued that positive associations are found only for particular measures and operationalizations of these constructs (Hackney & Sanders, 2003; Poloma & Pendleton, 1989). A recent meta-analysis of longitudinal studies reported that, out of eight religiosity/spirituality measures, only participation in public religious activities and the importance of religion were statistically significantly related to self-rated mental health, which was operationalized as distress, life satisfaction, well-being, and quality of life (Garssen et al., 2020).

Furthermore, the type of religiosity (i.e., intrinsic vs. extrinsic; positive vs. negative religious coping) and religious status (religious vs. uncertain) appear to moderate the relationship between religion and mental well-being (Smith et al., 2003; Villani et al., 2019). For instance, extrinsic religious orientation (i.e., when people primarily use their religious community as a social network, whereas personal religious beliefs are secondary) and negative religious coping (i.e., when people have internal religious guilt or doubts) have been shown to be negatively related to well-being (Abu-Raiya, 2013; Weber & Pargament, 2014). Yet other research suggests that it is precisely the social aspect of religious service attendance and congregational friendships that explains how religiosity is positively associated with life satisfaction (Lim & Putnam, 2010). Moreover, the direction of the religiosity—mental health relation remains unclear; while engaging in religious activities might make people happier, people with better mental health might also be more likely to engage in public, social events.

Additionally, there is large variability in the extent to which religion is ingrained in culture and social identity across the globe (Kelley & de Graaf, 1997; Ruiter & van Tubergen, 2009). Accordingly, when investigating the association between religiosity and well-being, it may be necessary to take into account the cultural norms related to religiosity within a society. Being religious may contribute to self-rated health and happiness when being religious is perceived to be a socially expected and desirable option (Diener et al., 2011; Ebert et al., 2020; Gebauer et al., 2017; Stavrova, 2015; Stavrova et al., 2013). This makes sense from the literature on person-culture fit (Dressler et al., 2007): a high person-culture fit indicates good agreement between one's personal values and beliefs and the beliefs that are shared by one's surrounding culture. A fruitful way to measure cultural norms is through the shared, intersubjective perception of the beliefs and attitudes that are prevalent in a society (Chiu et al., 2010; Zou et al., 2009). Intersubjective norms of religiosity, for instance, refer to the shared perception of the importance of religion within a society or culture. Rather than expressing the importance of religious beliefs and behaviors in one's own personal life, intersubjective norms of religiosity (henceforth: cultural norms of religiosity) uncover the perceived importance of religious beliefs and behaviors for the average person within a culture. Religious individuals may be more likely to benefit from being religious when their convictions and behaviors align with perceived cultural norms. For countries in which religion is more trivial or even stigmatized, the relation between religiosity and well-being may be absent or even reversed. Relatedly, in secular countries, religion might be practiced relatively often by minority groups, which has been shown to attenuate the positive association between religious involvement and well-being (Hayward & Elliott, 2014; Huijts & Kraaykamp, 2011; May & Smilde, 2016; Okulicz-Kozaryn, 2010).

1.2. A many-analysts approach

In the current project, we aim to shed light on the association between religion and well-being and the extent to which different theoretically- or methodologically-motivated analytic choices affect the results. To this end, we initiated a many-analysts project, in which several independent analysis teams analyze the same dataset in order to answer a specific research question (e.g., Bastiaansen et al., 2020; Boehm et al., 2018; Botvinik-Nezer et al., 2020; Silberzahn & Uhlmann, 2015; van Dongen et al., 2019). A many-analysts approach has been proposed as a way to mitigate the influence of individual-researcher biases (e.g., confirmation bias by the proponent of a theory or disconfirmation bias by the skeptic), especially since the analysis teams are not typically invested in the outcome. More generally, a many-analysts study is arguable less vulnerable to publication bias toward publishing only significant rather than null results, which may lower the (unconscious) tendency toward p-hacking by individual analysts. A many-analysts approach can balance out the effects of researcher bias while still allowing for expertize-based analytic decisions such as reasonable preprocessing steps, variable exclusion, and model specification. As such, it enables one to assess the robustness of outcomes and quantify variability based on theory-driven analysis decisions and plausible statistical models. Specifically, we believe that the more consistent the results from different analysis teams are, the more confident we can be in the conclusions we draw from the results. A many-analysts approach may be preferable to an exhaustive multiverse analysis (Steegen et al., 2016) that might simply include the full spectrum of options, including those that are theoretically and methodologically unrealistic.

The idea of inviting different analysis teams to answer the same research question using the same data is relatively novel (Silberzahn & Uhlmann, 2015; see Aczel et al., 2021 for general guidelines); we are aware of three papers in neuroscience (Botvinik-Nezer et al., 2020; Fillard et al., 2011; Maier-Hein et al., 2017), one in microeconomics (Huntington-Klein et al., 2021), and eight in psychology, three of which pertain to cognitive modeling (Boehm et al., 2018; Dutilh et al., 2019; Starns et al., 2019) while the remaining five are from other fields of psychology (Bastiaansen et al., 2020; Salganik et al., 2020; Schweinsberg et al., 2021; Silberzahn et al., 2018; van Dongen et al., 2019). Most similar to the current work are the projects that applied a many-analysts approach to perform statistical inference on the relation between two variables, such as skin color and red cards in soccer (Silberzahn et al., 2018), scientist gender and verbosity (Schweinsberg et al., 2021), or amygdala activity and stress (van Dongen et al., 2019). While the exact focus of previous many-analysts projects varied (e.g., experience sampling, fMRI preprocessing, predictive modeling, proof of the many-analysts concept), the take-home messages were rather consistent: all papers showed that different yet equally justifiable analytic choices result in very different outcomes, sometimes with statistically significant effects in opposite directions (e.g., Schweinsberg et al., 2021; Silberzahn et al., 2018). In addition, it has proved difficult to pinpoint the exact sources of variability due to the fact that analytic approaches differed in many respects simultaneously (e.g., exclusion criteria, inclusion of covariates etc.). Nevertheless, the outcomes of these previous projects suggest that choices of statistical model (Silberzahn et al., 2018), statistical framework (van Dongen et al., 2019), (pre)processing software (Botvinik-Nezer et al., 2020), and the variables themselves (Schweinsberg et al., 2021) exert substantial effects on the results and conclusions.

We believe a many-analysts approach is uniquely suited to address various concerns in the study of religion and well-being. First, the relation between religion and health has been researched for decades with hundreds of qualitative reports, cross-sectional and longitudinal studies, and even randomized controlled trials with religious/spiritual interventions for mental health issues (Captari et al., 2018; Harris et al., 2018; Koenig et al., 2020; Rosmarin et al., 2010). Yet new studies keep emerging (e.g., Chang et al., 2021; Luo & Chen, 2021; Simkin, 2020) and the debate seems far from settled (see for instance the recent special issue in the International Journal for the Psychology of Religion; van Elk, 2021). Second, both “religion” and “well-being” are broad and multifaceted constructs that are sensitive to different measures and operationalizations, which might result in both quantitatively and qualitatively different conclusions (Hackney & Sanders, 2003; Poloma & Pendleton, 1989). Third, the standard way to assess robustness of an effect or association is often through meta-analysis, but the fragmentation of the literature on the religion—health link and methodological heterogeneity between studies challenge the use and validity of meta-analyzes in this domain (Koenig et al., 2021). In general, meta-analyzes may suffer from several drawbacks such as publication bias and sensitivity to arbitrary methodological choices (e.g., different meta-analytic techniques can result in different conclusions; de Vrieze, 2018; van Elk et al., 2015). Moreover, the estimated effect sizes in meta-analyzes might be as much as three times larger than in preregistered multiple-site replication studies (Kvarven et al., 2020). Fourth, the discussion on the potential health-benefits of religion has been muddied by concerns about researcher interests and biases. That is, it has been argued that scholars of religion might be biased by their own (religious) beliefs (Ladd & Messick, 2016; Swigart et al., 2020; Wulff, 1998) or by the fact that a substantial amount of research in the science of religion is funded by religiously-oriented organizations such as the John Templeton Foundation (Bains, 2011; Wiebe, 2009).¹ Inviting independent analysts from various backgrounds including but not restricted to religious studies attenuates this potential concern. Moreover, in addition to quantifying variability, with a sufficiently large number of analysis teams one can also investigate factors that might explain observed variability, such as those related to theoretical or methodological expertize and prior beliefs (Aczel et al., 2021).²

In addition to the theoretical rationale for using a many-analysts approach to answer the research questions at hand, we also consider the current dataset particularly appropriate for such an approach. That is, the complexity of the data allows for many justifiable choices for the operationalization of the variables and the statistical approach to be employed. While the questions posed to the participants in the cross-cultural study could no longer be changed, the specific method of derivation for the religiosity and well-being scores was at the discretion of the many analysts. At the same time, the research questions and data structure (cross-sectional correlational data) were sufficiently intuitive and manageable to inspire many researchers in the fields of (social) psychology, religious studies, health science, and general methodology to propose an analysis.

Finally, we believe that our project involves a combination of elements that extend existing many-analysts work. First, we collected new data for this project with the aim to provide new evidence for the research questions of interest, as opposed to using an existing dataset that has been analyzed before. Second, we targeted both researchers interested in methodology and open science, as well as researchers from the field of the scientific study of religion and health to encourage both methodologically sound and theoretically relevant decisions (see the section “Analysis teams”). Third, in comparison to previous many-analysts projects in psychology, the current project includes a lot of teams (i.e., 120 vs. 4, 12, 14, 17, 27, 29, and 70 teams, though note that a machine learning project included 160 analyst teams; Salganik et al., 2020). Fourth, we applied a two-step procedure that ensured a purely confirmatory status of the analyzes: in stage 1, all teams first either completed a preregistration or specified an analysis pipeline based on a blinded version of the data. After submitting the plan to the OSF, teams received the real data and executed their planned analyzes in stage 2 (see Sarafoglou et al., 2022 for more details on and an empirical investigation of preregistration vs. data blinding based on the present data). Fifth, the many-analysts approach itself was preregistered prior to cross-cultural data collection (see osf.io/xg8y5), although the details of the processing and analysis of the many-analysts data were not preregistered.

1.3. The dataset

The dataset provided to the analysts featured data from $10,535$ participants from 24 countries collected in 2019. The data were collected as part of the cross-cultural religious replication project (see also Hoogeveen et al., 2021; Hoogeveen & van Elk, 2018). The dataset contained measures of religiosity, well-being, perceived cultural norms of religion, as well as some demographic items. The full dataset, the data documentation file, and original questionnaire can be found on the OSF project page (osf.io/qbdce/).

1.3.1. Participants

Participants were recruited from university student samples, from personal networks, and from (demographically representative) samples accessed by panel agencies and online platforms (MTurk, Kieskompas, Sojump, TurkPrime, Lancers, Qualtrics panels, Crowdpanel, and Prolific). Participants were compensated for participation by financial remuneration, the possibility for a reward through a raffle, course credits, or received no compensation. Everyone aged 18 years or above could participate.³ Participants were required to answer all multiple choice questions, and hence there were no missing data (except for 36 people who did not provide a numeric age and 995 people who chose not to answer the item on sexual satisfaction, as this was the only item for which participants were not required to provide an answer.) The countries were convenience-sampled (i.e., through personal networks), but were selected to cover six continents and include different ethnic and religious majorities. The final sample included individuals who identified as Christian ($31.2\text{\%}$), Muslim ($6.1\text{\%}$), Hindu ($2.9\text{\%}$), Buddhist ($2.0\text{\%}$), Jewish ($1.0\text{\%}$), or were part of another religious group ($2.9\text{\%}$). Finally, $53.9\text{\%}$ of participants did not identify with any religion. See Tables A1 and A2 in Appendix 1 for the full descriptive statistics of the dataset.

1.3.2. Measures

Personal religiosity was measured using nine standardized self-report items taken from the World Values Survey (WVS; World Values Survey, 2010), covering religious behaviors (institutionalized such as church attendance and private such as prayer/meditation), beliefs, identification, values, and denomination. The well-being measure consisted of 18 self-report items from the validated short version of Quality of Life scale, as used by the World Health Organization (WHOQOL-BREF; WHOQOL Group, 1998). Included items cover general health and well-being, as well as the domains of physical health, psychological health and social relationships. Specific items evaluated: the quality of life in general, and satisfaction of overall health (general); pain, energy, sleep, mobility, activities, dependence on medication, and work capability (physical domain); life enjoyment, concentration, self-esteem, body-image, negative feelings, and meaningfulness (psychological domain); as well as personal relationships, social support, and sexual satisfaction (social domain). In addition to the raw scores for each item, we also provided an overall mean, as well as three means per subscale, following the calculation instructions in the WHOQOL-BREF manual. Cultural norms of religiosity were measured with two items assessing participants' perception of the extent to which the average person in their country considers a religious lifestyle and belief in God/Gods/spirits important (Wan et al., 2007). Finally, demographics were measured at the individual level (i.e., age, gender, level of education, subjective socioeconomic status (SES), and ethnicity) whereas GDP per capita (current US$, World Bank Group, 2017), sample type (e.g., university students, online panels), and means of compensation (e.g., course credit, monetary reward) were determined at the country/sample level. Items were reverse-coded when applicable. Personal religiosity items were additionally rescaled to the 0-1 range to make them contribute equally to an average religiosity score since the items were measured on different scales (e.g., a 1–8 Likert scale or a “yes/no” item, which was coded as “no” $=0$ and “yes” $=1$).⁴ GDP was provided as a raw value as well as standardized at the country level.

2. Disclosures

2.1. Data, materials, and preregistration

At the start of this project we did not envision a particular statistical analysis to be executed across the reported results from the individual teams, and therefore we did not preregister any statistical inference procedure. However, at an earlier stage, we did preregister our own hypotheses regarding the research questions that were posed to the analysis teams (see osf.io/zyu8c/). This preregistration also anticipates the many-analysts approach, yet does not specify the exact details of the project. In this preregistration document, we indicated that the analysis teams would first receive a blinded version of the data, but we later decided that half of the teams would work with blinded data and the other half would write their own preregistration (see Sarafoglou et al., 2022). Note that we did not include our own estimated effect sizes in the results as shown below. Our results, however, do corroborate the overall pattern of results from the analysis teams. Interested readers can access our preregistered analysis of the research questions on the OSF (osf.io/vy8z7/).

All documents provided to the analysis teams (dataset, documentation, questionnaire), as well as the administered surveys, the anonymized raw and processed data (including relevant documentation), and the R code to conduct all analyzes (including all figures), can be found on the project page on the OSF (osf.io/vy8z7/). Identifying information (such as names, email-addresses, universities) was removed from all free-text answers. See also Table 2 for an overview of all resources.

Table 2. Overview of project stages and resources.

Process	Link
Stage 1
Recruitment and sign-up	osf.io/hpd6b
Pre-survey	osf.io/kgqze
Access to data documentation, questionnaire and either of:
(a) preregistration form	osf.io/a5ent
(b) blinded data	osf.io/ktvqw
Design analysis and upload plan	OSF team pages
Mid-survey	osf.io/kgqze
Stage 2
Access to data	osf.io/6njsy
Execute analysis and upload script (optional: + report)	OSF team pages
Post-survey	osf.io/kgqze
Lead team: summarize and write-up key results
Invite analysis teams to write commentary

Note: See osf.io/vy8z7 for an overview of all team pages.

2.2. Reporting

We report how we determined our sample size, all data exclusions, and all manipulations in the study. However, it should be noted that this project also involved an empirical evaluation of analysis blinding, which is reported in another paper (i.e., Sarafoglou et al., 2022). Here, we only describe measures relevant to the theoretical research questions and the many-analysts approach. The description of the remaining measures that were only used for the experimental analysis proposal manipulation can be found in Sarafoglou et al. (2022).

2.3. Ethical approval

The study was approved by the local ethics board of the University of Amsterdam (registration number: 2019-PML-12707). All participants were treated in accordance with the Declaration of Helsinki. See Appendix 1 for details on the ethical approval for the cross-cultural data collection.

3. Methods

3.1. Analysis teams

The analysis teams were recruited through advertisements in various newsletters and email lists (e.g., the International Association for the Psychology of Religion (IAPR), International Association for the Cognitive Science of Religion (IACSR), Society for Personality and Social Psychology (SPSP), and the Society for the Psychology of Religion and Spirituality (Div. 36 of the APA)), on social media platforms (i.e., blogposts and Twitter), and through the authors' personal network. We invited researchers of all career stages (i.e., from doctoral student to full professor). Teams were allowed to include graduate and undergraduate students in their teams as long as each team also included a PhD candidate or a more senior researcher. Initially, N = 173 teams signed up to participate in the many-analysts project. From those teams, N = 127 submitted an analysis plan and N = 120 completed the project. The members from each analysis team were offered co-authorship on the main manuscript. No individual researcher or team was excluded from the study.

The number of analysts per team ranged from 1 to 7, with most teams consisting of 1 (41%) or 2 (33%) analysts (median = 2). The different career stages and domains of expertize featured in the analysis teams are given in Table 1. In addition, Figure 1 shows the self-rated collective knowledge about the topic of religion and well-being and about methodology and statistics. As becomes evident, most of the analysis teams had more methodological and/or statistical expertize than substantive expertize; $80\text{\%}$ of the teams reported considerable expertize with regard to methods and statistics compared to $31\text{\%}$ with regard to religion and well-being, $19\text{\%}$ compared to $17\text{\%}$ was neutral, and $3\text{\%}$ compared to $50\text{\%}$ reported little to no knowledge, respectively.

Figure 1. Responses to the survey questions on self-rated topical and methodological knowledge. The top bar represents the teams' answers about their knowledge regarding religion and well-being and the bottom bar represents the teams' answers about their knowledge regarding methodology and statistics. For each item, the number to the left of the data bar (in brown/orange) indicates the percentage of teams that reported little to no knowledge. The number in the center of the data bar (in grey) indicates the percentage of teams that were neutral. The number to the right of the data bar (in green/blue) indicates the percentage of teams that reported (some) expertize.

Table 1. Career stages and domains of expertize featured in the 120 analysis teams.

	Percentage of teams
Career stages
Doctoral student	54 (45%)
Post-doc	45 (37.50%)
Assistant professor	32 (26.67%)
Associate professor	26 (21.67%)
Full professor	20 (16.67%)
Domains of expertize
Social psychology	43 (35.83%)
Cognition	28 (23.33%)
Methodology and statistics	25 (20.83%)
Religion and culture	25 (20.83%)
Psychology (other)	19 (15.83%)
Health	17 (14.17%)

Note: Teams may include multiple members of the same position and in the same domain.

3.2. Sampling plan

For a separate component of the project (see Sarafoglou et al., 2022), the preregistered sample size target was set to a minimum of 20 participating teams, which was based on the recruited analysis teams in the many-analysts project from Silberzahn et al. (2018). However, we did not set a maximum number of participating teams. The recruitment of analysis teams was ended on December 22, 2020.

3.3. Materials

3.3.1. Surveys

The analysts received three surveys, here referred to as the pre-survey, the mid-survey, and the post-survey. In the pre-survey, participating teams indicated the career stages and domains of expertize featured in their team, self-rated their (collective) theoretical and methodological knowledge (5-pt Likert scale), and anticipated the likelihood of the effects of interest (7-pt Likert scale). In the mid-survey, teams were asked about the experienced effort, frustration, workload in hours spent on the project, and the extent to which this workload was lower or higher than expected for the analysis planning phase (i.e., stage 1; 7-pt Likert scales). In the post-survey, the teams provided the results of their analyzes and again indicated their experiences during the analysis executing phase (i.e., stage 2). Specifically, per research question, teams were asked about their statistical approach, the operationalization of the independent variable(s) and dependent variable(s), included covariates, analytic sample size, (unit of) effect size, p-value or Bayes factor, and additional steps they took for the analysis. Furthermore, for both research questions, the teams gave a subjective conclusion about the evidence for the effect (i.e., “good evidence for a relation,” “ambiguous evidence,” or “good evidence against a relation”), about the practical meaningfulness/relevance of the effect (based on the data; “yes” or “no”), and indicated again the likelihood of the effects of interest (on a 7-pt Likert scale). Additionally, teams indicated the appropriateness of their statistical approach (7-pt Likert scale), the suitability of the dataset for answering each research question (7-pt Likert scale), and whether or not they deviated from their planned analysis. In case this last question was answered affirmatively, they specified with regard to which aspects they deviated (i.e., hypotheses, included variables, operationalization of the independent variable(s), operationalization of the dependent variable(s), exclusion criteria, statistical test, statistical model, direction of the effect). Finally, teams again reported the experienced effort, frustration, workload in hours and the extent to which this workload was lower or higher than expected for stage 2 (on 7-pt Likert scales).

3.4. Procedure

After signing up, participating teams received a document outlining the aim of the project, the timeline, a short theoretical background with respect to the research questions, and a description of the dataset. Then, after completing the pre-survey, teams could access the full data documentation, the questionnaire as presented to the participants of the cross-cultural study, and either a blinded version of the data or a preregistration template, depending on which condition they had been assigned to. Teams could then design their analysis and upload their documents on their own team page on the OSF (deadline: December 22nd, 2020). The project leaders “froze” the stage 1 documents and sent the link to the mid-survey. Upon completion of this survey, teams automatically received access to the real data. They could execute and upload their final analysis scripts on the OSF until February 28th, 2021. Teams were encouraged to also upload a document summarizing their results, but this was not mandatory. Finally, all teams completed the post-survey. See Table 2 for an overview of the procedure.

4. Results

Here, we report the key results of the project. Specifically, we evaluate the teams' reported effect sizes and their subjective conclusions about the research questions (i.e., the primary results). In addition, we provide descriptive results about the many-analysts aspect (i.e., the secondary results: variability in analytic approaches, included variables, and the teams' experiences across the two different stages). Finally, we assessed whether or not the reported effect sizes are related to subjective beliefs about the likelihood of the research questions.

4.1. Primary results

Teams could report any effect size metric of their choosing, but we noted that we preferred a beta coefficient (i.e., a fully standardized coefficient; z-scored predictors and outcomes) to allow for a comparison between teams. As we correctly anticipated that (1) most teams would conduct linear regression analyzes (see Table 3) and (2) both the (scale of the) independent and dependent variables might vary across teams, we considered a beta coefficient the most suitable effect size metric. Note that our request for beta coefficients as effect size metrics may have affected the teams' choice of statistical model and encouraged them to use regression models that generate beta coefficients. For teams that did not provide a (fully) standardized coefficient, we recalculated the beta based on the respective team's analysis script whenever possible. Specifically, for (multilevel) linear regression models we used the effectsize package or the jtools package to extract standardized coefficients in R. For analyzes in SPSS and non-standard models in R, we standardized the data manually prior to executing the analyzes. Finally, many teams reported multiple effect sizes, as they either separately considered multiple predictors (e.g., religious beliefs and religious behaviors) and/or multiple dependent variables (e.g., psychological well-being and physical well-being). In that case, we asked the teams to provide us with one primary effect size they considered most relevant to answer the research question or to select one randomly. In the Online Appendix, we additionally list (1) effect sizes for the different subscales of the well-being measure as reported by the teams and (2) effect sizes from teams that could not provide a beta coefficient (e.g., machine learning models).

Table 3. Analytic approaches taken by the analysis teams.

Analytic approach	Percentage of teams
Multilevel linear regression	45/128 (35.16%)
Linear regression	36/128 (28.12%)
Bayesian multilevel linear regression	7/128 (5.47%)
Structural equation model	6/128 (4.69%)
ANOVA	5/128 (3.91%)
T-test	4/128 (3.12%)
Bayesian linear regression	3/128 (2.34%)
Path analysis	3/128 (2.34%)
Bayesian multilevel ordinal regression	2/128 (1.56%)
Ordinal logistic regression	2/128 (1.56%)
ANCOVA	1/128 (0.78%)
Bayesian additive regression trees	1/128 (0.78%)
Bayesian ANOVA	1/128 (0.78%)
Bayesian multilevel structural equation model	1/128 (0.78%)
Correlation	1/128 (0.78%)
Machine learning	1/128 (0.78%)
Meta-analysis	1/128 (0.78%)
Mixed-effects ANOVA	1/128 (0.78%)
Moderated generalized linear regression	1/128 (0.78%)
Multilevel structural equation model	1/128 (0.78%)
Multiverse analysis	1/128 (0.78%)
Multiverse of multilevel linear regression	1/128 (0.78%)
Network analysis	1/128 (0.78%)
Non-linear regression	1/128 (0.78%)
Non-parametric partial correlation	1/128 (0.78%)

Note: Some teams reported multiple statistical approaches.

4.1.1. Research question 1: “do religious people self-report higher well-being?”

We were able to extract 99 beta coefficients from the results provided by the 120 teams that completed stage 2.⁵ As shown in Figure 2, the results are remarkably consistent: all 99 teams reported a positive beta value, and for all teams the $95\text{\%}$ confidence/credible interval excludes zero. The median reported beta is 0.120 and the median absolute deviation is 0.036. Furthermore, $88\text{\%}$ of the teams concluded that there is good evidence for a positive relation between religiosity and self-reported well-being. Notably, although the teams were almost unanimous in their evaluation of research question 1, only eight of the 99 teams reported combinations of effect sizes and confidence/credibility intervals that matched those from another team (i.e., four effect sizes were reported twice). Do note that in contrast to the unanimity in results based on the beta coefficients, out of the 21 teams for whom a beta coefficient could not be calculated, 3 teams reported evidence against the relation between religiosity and well-being: 2 teams used machine learning and found that none of the religiosity items contributed substantially to predicting well-being and 1 team used multilevel modeling and reported unstandardized gamma-weights for within- and between-country effects of religiosity whose confidence intervals included zero (see the Online Appendix).

Figure 2. Beta coefficients for the effect of religiosity on self-reported well-being (research question 1) with 95% confidence or credible intervals. Green/blue points indicate effect sizes of teams that subjectively concluded that there is good evidence for a positive relation between individual religiosity and self-reported well-being, grey points indicate effect sizes of teams that subjectively concluded that the evidence is ambiguous, and brown/orange points indicate effect sizes of teams that subjectively concluded that there is good evidence against a positive relation between individual religiosity and self-reported well-being. The betas are ordered from smallest to largest.

Figure 3 displays the average prior and final beliefs about the likelihood of the hypothesis. Researchers' prior beliefs about religiosity being positively related to self-reported well-being were already high (M = 4.90 on the 7-point Likert scale), but were raised further after them having conducted the analysis (M = 5.49 on the 7-point Likert scale). Specifically, before seeing the data, $72\text{\%}$ of the teams considered it likely that religiosity is related to higher self-reported well-being. This percentage increased to $85\text{\%}$ after having seen the data, while $11\text{\%}$ were neutral and $3\text{\%}$ considered it unlikely. Finally, $75\text{\%}$ of teams indicated the relation to be relevant or meaningful based on these data.

Figure 3. Responses to the survey questions about the likelihood of hypothesis 1. The left side of the figure shows the change in beliefs for each analysis team. Fifty percent of the teams considered the hypothesis somewhat more likely after having analyzed the data than prior to seeing the data, 18% considered the hypothesis less likely after having analyzed the data, and 32% did not change their beliefs. Likelihood was measured on a 7-point Likert scale ranging from “very unlikely” to “very likely.” Points are jittered to enhance visibility. The right side of the figure shows the distribution of the Likert response options before and after having conducted the analyzes. The number at the top of the data bar (in green/blue) indicates the percentage of teams that considered the hypothesis (very) likely, the number in the center of the data bar (in grey) indicates the percentage of teams that were neutral, and the number at the bottom of the data bar (in brown/orange) indicates the percentage of teams that considered the hypothesis (very) unlikely.

4.1.2. Research question 2: “does the relation between religiosity and self-reported well-being depend on perceived cultural norms of religion?”

Out of the 120 teams who completed stage 2 we were able to extract 101 beta coefficients for research question 2. As shown in Figure 4 the results for research question 2 are more variable than for research question 1; 97 out of 101 teams reported a positive beta value and for 66 teams ($65\text{\%}$) the confidence/credible interval excluded zero. The median reported effect size is 0.039 and the median absolute deviation is 0.022. Furthermore, $54\text{\%}$ of the teams concluded that there is good evidence for an effect of cultural norms on the relation between religiosity and self-reported well-being. Again, most reported effect sizes were unique; only 3 out of the 101 reported combination of effect size and confidence/credible intervals appeared twice.

Figure 4. Beta coefficients for the effect of cultural norms of the relation between religiosity and self-reported well-being (research question 2) with 95% confidence or credible intervals. Green/blue points indicate effect sizes of teams that subjectively concluded that there is good evidence for the hypothesis that the relation between individual religiosity and self-reported well-being depends on the perceived cultural norms of religion, grey points indicate effect sizes of teams that subjectively concluded that the evidence is ambiguous, and brown/orange points indicate effect sizes of teams that subjectively concluded that there is good evidence against the hypothesis that the relation between individual religiosity and self-reported well-being depends on the perceived cultural norms of religion. The betas are ordered from smallest to largest.

Figure 5 shows the researchers' average prior and final beliefs about the likelihood of the second hypothesis. As for research question 1, prior beliefs about the hypothesis were rather high. However, in contrast to research question 1, conducting the analysis lowered beliefs about the likelihood of hypothesis 2. Specifically, before seeing the data, $71\text{\%}$ of the teams considered it likely that the relation between religiosity and self-reported well-being depends on perceived cultural norms of religion. This percentage dropped to $54\text{\%}$ after having seen the data, while $19\text{\%}$ were neutral and $27\text{\%}$ considered it unlikely. Finally, only about half of the teams ($49\text{\%}$) indicated the effect of cultural norms to be relevant or meaningful based on these data.

Figure 5. Responses to the survey questions about the likelihood of hypothesis 2. The left side of the figure shows the change in beliefs for each analysis team. Twenty-seven percent of the teams considered the hypothesis somewhat more likely after having analyzed the data than prior to seeing the data, 45% considered the hypothesis less likely having analyzed the data, and 28% did not change their beliefs. Likelihood was measured on a 7-point Likert scale ranging from “very unlikely” to “very likely.” Points are jittered to enhance visibility. The right side of the figure shows the distribution of the Likert response options before and after having conducted the analyzes. The number at the top of the data bar indicates the percentage of teams that considered the hypothesis (very) likely, the number in the center of the data bar (in grey) indicates the percentage of teams that were neutral, and the number at the bottom of the data bar (in brown/orange) indicates the percentage of teams that considered the hypothesis (very) unlikely.

4.2. Secondary results

In addition to evaluating the overall results for the two main research questions, we also assessed perceived suitability of the data and analytic approaches, variability in analytical approaches (i.e., statistical models), variable inclusion, and teams' experiences during the two stages of the project.

4.2.1. Perceived suitability of dataset

At the end of the project, all teams reported how suitable they found the current dataset for answering the research questions. As shown at the top of Figure 6, most teams considered the data (very) suitable for answering the research questions: for research question 1, $86\text{\%}$ found the data suitable, $8\text{\%}$ neutral, and $6\text{\%}$ unsuitable; for research question 2, $70\text{\%}$ found the data suitable, $19\text{\%}$ neutral, and $11\text{\%}$ unsuitable.

Figure 6. Responses to the survey questions about the suitability of the dataset for answering the research questions (top) and the teams' confidence in their analytic approach (bottom). For question 1, the top bar represents the teams' answers with respect to research question 1 and the bottom bar represents the teams' answers for research question 2. For each item, the number to the left of the data bar (in brown/orange) indicates the percentage of teams that considered the data (very) unsuitable/were not (at all) confident in their approach. The number in the center of the data bar (in grey) indicates the percentage of teams that were neutral. The number to the right of the data bar (in green/blue) indicates the percentage of teams that considered the data (very) suitable/were (very) confident in their approach.

4.2.2. Analytic approaches

Table 3 displays the different statistical approaches used in the project, as well as the percentage of teams that employed the respective approach. While a total of 25 different statistical methods was mentioned, (multilevel) linear regression was clearly the dominant approach. Specifically, $34\text{\%}$ of the teams used linear regression, another $45\text{\%}$ used multilevel linear regression, and the remaining $21\text{\%}$ used a different approach.

In general, teams were confident that their chosen statistical approach was appropriate for analyzing the research questions; as shown at the bottom of Figure 6, $89\text{\%}$ of the teams indicated to be (very) confident, $4\text{\%}$ was neutral, and $7\text{\%}$ was not (at all) confident.⁶

4.2.3. Variable inclusion

For each team we coded which of the items provided in the dataset were included as (1) dependent variable, (2) independent variable, and (3) covariates in the analysis for each research question.⁷

4.2.3.1 Dependent variable

The subjective well-being measure consisted of three subscales (psychological, physical, social), as well as two general items. In the dataset, we provided responses for all 18 individual items as well as an overall mean and one mean for each of the three subscales. Teams could decide to either use any of the provided averages or combine specific items themselves (e.g., take the mean, median, sum). In addition, some teams conducted a factor analysis and used one or multiple extracted factors as the dependent variable. In this case, we coded which items were used as input for the factor analysis. Figure 7 shows the included items as dependent variable aggregated over all teams for research question 1 and research question 2. For research question 1, the most frequently used items are enjoying life and meaningfulness (included by over $43\text{\%}$ of the teams). Note that all but four teams used the same dependent variable for research question 1 and 2.⁸ In Appendix 2, we show the included items separately for each team.

Figure 7. Items included as dependent variables for research question 1 (on the left) and research question 2 (on the right). Note that the averages for the well-being subscales (“Mean Psychological,” “Mean Social,” “Mean Physical”), as well as the overall average (“Mean Overall”) were provided by the MARP team.

4.2.3.2 Independent variable

The religiosity measure consisted of 9 primary items on response scales ranging from dichotomous to 8-points and the cultural norms of religiosity measure consisted of two items on a 5-point scale. Averages were not provided in the dataset, but could be created by the teams themselves. Figure 8 shows the included items as independent variable aggregated over all teams for research question 1 and research question 2. In Appendix 2, we show the included items separately for each team.

Figure 8. Items included as independent variables for research question 1 (on the left) and research question 2 (on the right). Variables indicated as “external” refer to variables that are based on data not provided by the MARP team.

For research question 1 (i.e., the relation between religiosity and self-reported well-being), over $75\text{\%}$ of the teams operationalized the independent variable by including the items frequency of service attendance, belief in God/Gods, frequency of prayer, belief in afterlife, personal importance of a religious lifestyle, or personal importance of belief in God. The remaining three religiosity items were used less frequently: $70\text{\%}$ of the teams included the item religious status (religious/not religious/atheist) and spirituality, while only $50\text{\%}$ included religious membership.

For research question 2 (i.e., the effect of perceived cultural norms on the relation between religiosity and self-reported well-being), all but four teams used the interaction term between their chosen religiosity measure and their chosen cultural norms measure as the independent variable.⁹ More teams operationalized cultural norms using the item importance of a religious lifestyle in their country ($93\text{\%}$) than importance of belief in God/Gods in their country ($89\text{\%}$). Here again, over $75\text{\%}$ of the teams operationalized the independent variable by including the items frequency of service attendance, belief in God, frequency of prayer, belief in afterlife, personal importance of a religious lifestyle, or personal importance of belief in God, whereas the items religious status (religious/not religious/atheist) and spirituality were included by about $70\text{\%}$ and $68\text{\%}$ of the teams, respectively; only $52\text{\%}$ of the teams included religious membership. Note that almost all teams used the same religiosity measure for research question 1 and research question 2.

4.2.3.3 Covariates

Teams were free to include as covariates in their models any of the measured demographic variables (e.g., age, socio-economic status), country-level variables (e.g., gross domestic product—GDP) or sample characteristics (e.g., general public or student sample, means of compensation). Figure 9 displays the included items as covariates aggregated over all teams for research question 1 and research question 2. The most frequently included covariates are age ($59\text{\%}$), socio-economic status ($55\text{\%}$), gender ($53\text{\%}$), and education ($50\text{\%}$). Note that per team the choice of covariates was largely equal across research questions, with the exception that the cultural norms items were occasionally added as covariates for research question 1 while they were part of the independent variable for research question 2.

Figure 9. Items included as covariates for research question 1 (on the left) and research question 2 (on the right). Variables indicated as “external” refer to covariates that are based on data not provided by the MARP team.

4.2.4. Teams' experiences

Although most teams indicated that effort was (very) high, the majority also reported that frustration was (very) low and that they spent as much time as anticipated (see Figure 10). That is, in stage 1, $55\text{\%}$ of the teams reported (very) high effort, $17\text{\%}$ were neutral, and $28\text{\%}$ reported (very) low effort. For stage 2, $48\text{\%}$ of the teams reported (very) high effort, $18\text{\%}$ were neutral, and $34\text{\%}$ reported (very) low effort. In stage 1, $17\text{\%}$ of the teams reported (very) high frustration, $23\text{\%}$ were neutral, and $60\text{\%}$ reported (very) low frustration. In stage 2, $18\text{\%}$ of the teams reported (very) high frustration, $17\text{\%}$ were neutral, and $65\text{\%}$ reported (very) low frustration. The median time spent on the analyzes was 8 hours for both stages, although the range was quite wide: 1 to 80 hours for stage 1 and 30 minutes to 140 hours for stage 2. Most teams anticipated as much time as they spent: $51\text{\%}$ for stage 1 and $52\text{\%}$ for stage 2. In stage 1, $36\text{\%}$ spent (much) more time than anticipated and $13\text{\%}$ spent (much) less time. In stage 2, $33\text{\%}$ spent (much) more time than anticipated and $15\text{\%}$ spent (much) less time.

Figure 10. Responses to the survey questions about effort (top), frustration (middle), and workload (bottom). For each question, the top bar represents the teams' answers about stage 1 (planning) and the bottom bar represents the teams' answers about stage 2 (executing). For each item, the number to the left of the data bar (in brown/orange) indicates the percentage of teams that considered effort/frustration/workload (very) low. The number in the center of the data bar (in grey) indicates the percentage of teams that were neutral. The number to the right of the data bar (in green/blue) indicates the percentage of teams that considered effort/frustration/workload (very) high.

4.2.5. Correlation between effect sizes and subjective beliefs

Following Silberzahn et al. (2018) we explored whether the reported effect sizes were positively related to subjective beliefs about the plausibility of the research question before and after analyzing the data. This hypothesis was tested against the null-hypothesis that there is no relation between reported effect sizes and subjective beliefs. As the subjective beliefs were measured on a 7-point Likert scale, we used a rank-based Spearman correlation test with a Uniform$[0,1]$ prior (van Doorn et al., 2020).

For research question 1, we obtained strong evidence against a positive relation between prior beliefs about the plausibility of the research question and the reported effect sizes: ${\mathrm{BF}}_{+0}=0.03$; ${\mathrm{BF}}_{0+}=30.34$, ${\rho }_s=-0.21$, 95% credible interval [−0.37, −0.04]. In addition, we found moderate evidence against a positive relation between posterior beliefs about the plausibility of the research question and the reported effect sizes: ${\mathrm{BF}}_{+0}=0.31$; ${\mathrm{BF}}_{0+}=3.18$, ${\rho }_s=0.10$, 95% credible interval [−0.08, 0.27].

For research question 2, we found moderate evidence against a positive relation between prior beliefs about the plausibility of the research question and the reported effect sizes: ${\mathrm{BF}}_{+0}=0.12$; ${\mathrm{BF}}_{0+}=8.55$, ${\rho }_s=0.01$, 95% credible interval [−0.16, 0.18]. For the posterior beliefs, however, we obtained strong evidence in favor of a positive relation between posterior beliefs about the plausibility of the research question and the reported effect sizes: ${\mathrm{BF}}_{+0}=67.39$, ${\rho }_s=0.33$, 95% credible interval [0.15, 0.46].

To further investigate changes in belief over the course of the project, we assessed the correlation between the reported effect sizes and the change in belief (i.e., the difference between posterior and prior beliefs for both research questions). For research question 1, there was basically no evidence for or against a positive relation between effect size and change in belief: ${\mathrm{BF}}_{+0}=1.81$, ${\rho }_s=0.18$, 95% credible interval [0.01, 0.33]. For research question 2 on the other hand, we obtained moderate evidence that effect sizes were positively related to change in subjective belief about the plausibility of the hypothesis: ${\mathrm{BF}}_{+0}=9.88$, ${\rho }_s=0.24$, 95% credible interval [0.07, 0.39].

These results regarding prior beliefs provide no indication that expectations and confirmation bias influenced the teams' results. For the posterior beliefs, on the other hand, it seems that the teams updated their beliefs about the plausibility of research question 2 based on the results of their analyzes. Note, however, that based on the scatterplot in Figure 11(D), we should not put too much weight on this finding, as it may be partly driven by two outliers. For research question 1, the updating of beliefs may not have happened because prior beliefs about research question 1 were already in line with the outcomes, i.e., most teams expected and reported evidence for a positive relation between religiosity and well-being, with little variation between teams.

Figure 11. Reported effect sizes (beta coefficients) and subjective beliefs about the likelihood of the hypothesis. A. shows the relation between effect size and prior beliefs for research question 1, B. shows the relation between effect size and final beliefs for research question 1, C. shows the relation between effect size and prior beliefs for research question 2, and D. shows the relation between effect size and final beliefs for research question 2. Points are jittered on the x-axis to enhance visibility. The dashed line represents an effect size of 0. The data are separated by subjective evaluation of the evidence; green/blue points reflect the conclusion that there is good evidence for the hypothesis, grey points reflect the conclusion that the evidence is ambiguous, and brown/orange points indicate the conclusion that there is good evidence against the hypothesis. Histograms at the top represent the distribution of subjective beliefs and the density plots on the right represent the distribution of reported effect sizes.

Finally, we assessed whether reported effect sizes were related to self-reported expertize. Here, we used a Uniform$[-1,1]$ prior and an undirected test. This hypothesis was tested against the null-hypothesis that reported effect sizes and self-reported expertize were not related. For research question 1, we found moderate evidence against a correlation between effect sizes and methodological knowledge (${\mathrm{BF}}_{10}=0.13$; ${\mathrm{BF}}_{01}=7.80$, ${\rho }_s=0.03$, 95% credible interval [−0.17, 0.21]) and weak evidence against a correlation between effect sizes and theoretical knowledge (${\mathrm{BF}}_{10}=0.48$; ${\mathrm{BF}}_{01}=2.09$, ${\rho }_s=-0.16$, 95% credible interval [−0.31, 0.03]). For research question 2, we again obtained moderate evidence against a relation between effect sizes and methodological knowledge (${\mathrm{BF}}_{10}=0.12$; ${\mathrm{BF}}_{01}=8.00$, ${\rho }_s=0.02$, 95% credible interval [−0.17, 0.20]) and moderate evidence against a correlation between effect sizes and theoretical knowledge (${\mathrm{BF}}_{10}=0.16$; ${\mathrm{BF}}_{01}=6.41$, ${\rho }_s=-0.08$, 95% credible interval [−0.24, 0.09]). See Figure 12 for scatterplots of the data.

Figure 12. Reported effect sizes (beta coefficients) and self-reported team expertize. A. shows the relation between effect size for research question 1 and methodological knowledge, B. shows the relation between effect size for research question 1 and theoretical knowledge, C. shows the relation between effect size and for research question 2 and methodological knowledge, and D. shows the relation between effect size for research question 2 and theoretical knowledge. Points are jittered on the x-axis to enhance visibility. The dashed line represents an effect size of 0. The data are separated by subjective evaluation of the evidence; green/blue points reflect the conclusion that there is good evidence for the hypothesis, grey points reflect the conclusion that the evidence is ambiguous, and brown/orange points indicate the conclusion that there is good evidence against the hypothesis. Histograms at the top represent the distribution of reported expertize and the density plots on the right represent the distribution of reported effect sizes.

5. Summary

In the current project, 120 analysis teams were given a large cross-cultural dataset ($N=10,535$, 24 countries) in order to investigate two research questions: (1) “Do religious people self-report higher well-being?” and (2) “Does the relation between religiosity and self-reported well-being depend on perceived cultural norms of religion?.” In a two-stage procedure, the teams first proposed an analysis and then executed their planned analysis on the data.

Perhaps surprisingly in light of previous many-analysts projects, results were fairly consistent across teams. For research question 1 on the relation between religiosity and self-reported well-being, all but three teams reported a positive effect size and confidence/credible intervals that exclude zero. For research question 2, the results were somewhat more variable: $95\text{\%}$ of the teams reported a positive effect size for the moderating influence of cultural norms of religion on the association between religiosity and self-reported well-being, with $65\text{\%}$ of the confidence/credible intervals excluding zero. While most teams used (multilevel) linear regression, there was considerable variability in the choice of items used to construct the independent variable, the dependent variable, and the included covariates.

A further discussion of these results including limitations and broader implications, as well as a reflection on the many-analysts approach is covered in the closing article (Hoogeveen et al., 2022). There, we also address the commentaries written by some of the analysis teams.

Author contributions

Contributorship was documented with CRediT taxonomy using tenzing (Holcombe et al., 2020). Conceptualization: S.H., A.S., M.v.E., and E.-J.W.Data curation: S.H. and A.S.Formal analysis: S.H., A.S., B.A., Y. Aditya, A.J.A., P.J.A., S. Alzahawi, Y. Amir, F.-V.A., O.K.A., Q.D.A., A. Baimel, M.B.-I., M. Balsamo, S.B., F.B., M. Becerra, B.B., J. Beitner, T. Bendixen, J.B.B., M.I.B., J. Billingsley, T. Bortolini, H.B., A. Bret, F.L.B., J. Brown, C.C.B., J. Buczny, J. Bulbulia, S. Caballero, L.C., C.L.C., M.E.G.V.C., S.J.C., S. Claessens, M.C.P., A.B.C., D.L.C., S. Czoschke, C.C., E.D.D.U., Ö.D., A.D.R., H.D., J.D.R., Y.A.d.V., K.K.D., B.J.D., D.J.D., J.K.D., T.D., L.D., M.D., Y.D., T.E., P.A.E., A.E., C.T.E., S. Farahmand, H.F., M.F., A.A.F., K.F., R.F., D.F.-T., Z.F., S. Frick, L.K.F., D.G., E. Gerdin, L.G., O.G., E. Gielens, V.G., H.R.H., I.H., P.H.P.H., C.E.H., B.C.H., L.E.H., M.I., H.I., M.L.I., C.-G.I., O.I., D.I., B.J., K.A.J., J.J., J.A.K., K.K.H., E.K., B.A.K., L.A.K., S. Kelchtermans, J.M.K., R.A.K., B.K., M.L.K., M.K., D.K., J.K., S. Kritzler, A.-M.K., T.K., T.L.L., R.L., G.A.L.F., A.L., B.J.L., R.F.L., P.L., J.L., P.Ł., A.L.L., E.M., G.M.M.-R., M. Maier, D.R.M., N.M., M. Martinie, I.M., S.E.M., A.L.M., P. McAleer, T.M., M. McCullough, C.M.M., A.A.M., K.K.M., B. Mercier, P. Mitkidis, B. Monin, J.W.M., D.M., J. Morgan, J. Murphy, G.M., C.N., T.N., L.N., N.H., G.N., P.N., A.N., M.B.N., A.O.-C., L.O., Y.G.P., J.O.P., H.I.P., H.P., H.K.P., M. Pinus, S.P., V.P., M. Porubanova, M.J.P., J.M.P., M.A.P., J.P., C.P., B.G.P., J.Q.P., M.L.R., G.R., A. Roberts, L.M.R.L., R.M.R., P.R., N.R., S.-M.K.S., J.Y.S., C. Schaumans, B. Schivinski, M.C.S., S.A.S., M. Schnuerch, M.R.S., V.S., S. Sebben, S.C.S., B. Seryczyńska, U.S., M. Simsek, W.W.A.S., E.R.S., W.J.S., M. Späth, C. Spörlein, W.S., A.H.S., S. Stuber, J.S., C. Suwartono, S. Syropoulos, B. Szaszi, P.S., L.T., R.T.T., B.T., C.M.T., J.T., S.D.T., A.-M.U., R.C.M.V.A., M.A.L.M.v.A., P.V.C., O.R.V.d.A., I.V.d.C., J.V.d.N., N.N.N.v.D., C.J.V.L., V.v.M., D.v.R., C.J.J.v.Z., L.A.V., B. Većkalov, B. Verschuere, M.V., F.V., A.V., V.V., L.V.D.E.V., S. Watanabe, C.J.M.W., K.W., S. Wiechert, Z.W., M.W., C.V.O.W., D.W., X.Y., D.J.Y., O.Y., N.Z., Y.Z., and J.Z.Funding acquisition: A.S., M.v.E., E.-J.W., S. Altay, N.L., R.M., and R.M.R.Investigation: S.H., A.S., M.v.E., E.-J.W., S. Altay, T. Bendixen, R.B., K.H., R.M., L.Q., A. Rabelo, J.E.R., R.M.R., R.W., and D.X.Methodology: S.H., A.S., M.v.E., and E.-J.W.Project administration: S.H., A.S., M.v.E., and E.-J.W.Supervision: M.v.E. and E.-J.W.Validation: S.H. and A.S.Visualization: S.H., A.S., and P.J.A.Writing—original draft: S.H., A.S., M.v.E., and E.-J.W.Writing—review & editing: P.A.E., P.H.P.H., R.M., C.M.M., J. Murphy, T.N., J.E.R., R.M.R., S.C.S., B.T., R.C.M.V.A., M.A.L.M.v.A., and C.J.M.W.

Acknowledgments

We thank Arik Cheshin, Claudio Gentili, Raluca Georgescu, Christopher Kavanagh, Alejandra Neely, Hugh Turpin, and Filip Uzarevic for their assistance in creating the material and collecting the cross-cultural data.

Disclosure statement

The authors declare that there were no conflicts of interest with respect to the authorship or the publication of this article.

Additional information

Funding

Neil Levy and Robert M. Ross were supported by Australian Research Council (grant number DP180102384); Ryan McKay was supported by Cogito Foundation [grant number R10917]; Sacha Altay was supported by French Agence Nationale de la Recherche (reference 17-EURE-0017 FrontCog and 10-IDEX-0001-02 PSL); Emily Gerdin was supported by National Science Foundation Graduate Research Fellowship (GR100544) [grant number DGE-2139841]; Eric-Jan Wagenmakers was supported by Netherlands Organisation for Scientific Research [grant number 016.Vici.170.083]; Don van Ravenzwaaij was supported by Netherlands Organisation for Scientific Research [grant number 016.Vidi.188.001]; Alexandra Sarafoglou was supported by Netherlands Organisation for Scientific Research [grant number 406-17-568]; Michiel van Elk was supported by John Templeton Foundation [grant number 60663]; James O. Pawelski and Louis Tay were supported by Templeton Religion Trust (TRT 0154). Marcel R. Schreiner, Susanne Frick, Julian Quevedo Pütter and Marcel C. Schmitt were supported by the German Research Foundation [grant number GRK 2277].

Notes

1 Ironically, so is the present project.

2 Note that we acknowledge that another important problem in the literature on religion and well-being concerns the issue of causality. However, as our project uses non-experimental cross-sectional data, this issue cannot immediately be addressed in the current study (but see Grosz et al., 2020; Rohrer, 2018 for a perspective on causal inference in non-experimental studies).

3 Note that we did not exclude the 19 participants who indicated they were younger than 18 (but some of the analysis teams did exclude these participants).

4 When teams indicated that they preferred the raw data, we provided the function to back-transform the data.

5 One team misinterpreted the scoring of the items and hence miscoded the direction of the effect. As they subsequently also based their subjective conclusions on the incorrect results, we excluded the reported effect sizes, subjective evaluation, and prior+final beliefs about the likelihood of the hypotheses for this team.

6 Note that out of the 8 teams reporting not being confident, 2 did not submit a final analysis and 2 did not provide a usable effect size.

7 Please see the document “variable mapping” on the OSF (osf.io/qbdce/) for how the items correspond to the item names in the datafile.

8 Two of the four teams that did not use the same dependent variable for research question 1 and 2 only conducted an analysis for research question 1.

9 The four teams that did not use an interaction in their evaluation of research question 2 either used the main effect of cultural norms on well-being or the main effect of religiosity on well-being (while controlling for cultural norms).

AppendicesAppendix 1.

Descriptives of the dataset

A.1. Ethical approval cross-cultural data collection

Data collection for the cross-cultural study was approved by the local ethics committee at the Psychology Department of the University of Amsterdam (Project #2018-SP-9713). Additional approval was obtained from local IRBs at the Adolfo Ibáñez University (Chile), the Babes-Bolyai University (Romania), the James Cook University (Singapore), Royal Holloway, University of London (UK), the University of Connecticut (US), and the Max Planck Society, as well as the Senate Department for Education, Youth and Family from the Ministry of Education in Berlin (Germany). All participants were treated in accordance with the Declaration of Helsinki.

Table A1. Descriptive statistics of the cross-cultural data.

Country	N	Age (SD)	Women	Well-being	Religiosity	Cultural norms
Australia	482	47.8 (15.9)	47.9%	3.7	0.53	0.46
Belgium	325	34.9 (13.4)	54.8%	3.8	0.27	0.22
Brazil	406	29.1 (10.8)	73.2%	3.5	0.51	0.61
Canada	351	33.2 (10.5)	52.4%	3.6	0.30	0.29
Chile	308	30.8 (9.9)	59.1%	3.7	0.36	0.44
China	420	32.1 (8.5)	55.0%	3.6	0.33	0.31
Croatia	311	28.0 (6.8)	77.8%	3.8	0.44	0.58
Denmark	420	27.9 (10.4)	70.7%	3.8	0.29	0.19
France	409	40.5 (12.8)	63.8%	3.6	0.32	0.30
Germany	1,306	27.6 (9.1)	62.2%	3.8	0.36	0.35
India	454	30.4 (6.4)	37.9%	3.8	0.74	0.70
Ireland	434	42.6 (15.0)	51.8%	3.6	0.49	0.46
Israel	514	28.1 (10.5)	73.0%	3.8	0.36	0.49
Italy	347	27.2 (8.3)	50.1%	3.5	0.27	0.47
Japan	427	40.5 (10.0)	43.6%	3.0	0.29	0.28
Lithuania	297	24.1 (6.9)	82.5%	3.5	0.37	0.43
Morocco	397	33.9 (13.1)	16.1%	3.6	0.71	0.75
Netherlands	499	58.0 (14.7)	25.1%	3.9	0.29	0.29
Romania	580	25.0 (8.2)	84.1%	3.8	0.59	0.59
Singapore	327	22.2 (3.5)	62.4%	3.5	0.46	0.39
Spain	341	41.9 (13.9)	31.4%	3.7	0.25	0.28
Turkey	367	39.3 (11.2)	24.3%	3.7	0.34	0.64
UK	400	36.2 (12.7)	65.8%	3.5	0.25	0.20
US	413	35.6 (14.4)	50.6%	3.7	0.48	0.46
Total	10,535	33.8 (13.9)	55.4%	3.7	0.40	0.42

Note: Well-being refers to the average of self-rated general (2 items), physical (7 items), psychological (6 items), and social (3 items) health, measured on a 5-point Likert scale. Religiosity refers to the average self-reported level of individual religiosity based on 9 items. Cultural norms refers to the average perceived descriptive norm of religiosity in one's country based on 2 items. Religiosity and cultural norms are transformed on a 0-1 scale.

Table A2. Religious denomination in the cross-cultural data.

	Religious group
Country	Christian	Muslim	Hindu	Buddhist	Jewish	Other	None
Australia	43.2%	5.2%	0.6%	0.6%	0.4%	3.7%	46.3%
Belgium	28.0%	2.8%	0.0%	0.6%	0.3%	1.2%	67.1%
Brazil	30.0%	0.0%	0.0%	1.0%	0.2%	14.5%	54.2%
Canada	26.5%	1.1%	0.9%	1.1%	2.0%	1.4%	67.0%
Chile	25.3%	0.0%	0.3%	1.6%	3.2%	2.9%	66.6%
China	4.5%	0.2%	0.0%	11.2%	0.0%	1.0%	83.1%
Croatia	54.3%	0.3%	0.0%	0.6%	0.6%	0.6%	43.4%
Denmark	36.2%	2.1%	0.0%	0.0%	0.0%	0.0%	61.7%
France	38.4%	6.1%	0.0%	0.2%	0.0%	1.2%	54.0%
Germany	54.1%	3.4%	0.1%	0.1%	0.3%	1.2%	40.8%
India	14.1%	3.3%	60.1%	0.4%	0.2%	0.9%	20.9%
Ireland	51.6%	1.6%	0.2%	0.0%	0.2%	3.7%	42.6%
Israel	2.5%	3.1%	0.0%	0.0%	11.9%	1.9%	80.5%
Italy	17.9%	0.3%	0.0%	0.9%	0.0%	0.0%	81.0%
Japan	0.9%	0.2%	0.0%	15.2%	0.0%	1.2%	82.4%
Lithuania	39.1%	0.0%	0.3%	0.0%	0.0%	0.7%	59.9%
Morocco	0.5%	77.1%	0.0%	0.0%	0.3%	3.8%	18.4%
Netherlands	24.6%	0.0%	0.0%	0.0%	0.6%	6.2%	68.5%
Romania	74.0%	0.2%	0.0%	0.2%	0.2%	5.2%	20.3%
Singapore	17.1%	4.9%	4.3%	19.6%	0.0%	8.6%	45.6%
Spain	40.5%	0.0%	0.0%	0.0%	0.0%	1.2%	58.4%
Turkey	0.0%	41.1%	0.0%	0.0%	0.3%	4.6%	54.0%
UK	22.2%	0.5%	0.8%	0.5%	0.8%	1.0%	74.2%
US	41.6%	1.2%	0.5%	0.5%	3.1%	5.3%	47.7%
Total	31.2%	6.1%	2.9%	2.0%	1.1%	2.9%	53.9%

Note: Percentage of people indicating to be member of the respective religious groups. Note that the response options were adjusted and further specified per country.

Appendix 2.

Variable inclusion

Here we show the variable inclusion for each team; Table A3 displays the included items for the dependent variable for research question 1 and Table A4 displays the included items for the dependent variable for research question 2. Table A5 displays the included items for the independent variable for research question 1 and Table A6 displays the included items for the independent variable for research question 2. Table A7 displays the included items as covariates for research question 1 and Table A8 displays the included items as covariates for research question 2.

Table A3. Items included as dependent variables for research question 1 by each team.

Team	Pain	Medication	Energy	Mobility	Sleep	Activities	Work ability	Enjoy life	Meaningfulness	Concentration	Satisfaction appearance	Self esteem	Negative affect	Relationships	Social support	Sexual satisfaction	Quality life	Satisfaction health	Mean physical	Mean psychological	Mean social	Mean overall	Counts
3	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X							16
5	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X					18
6													X										1
7																			X	X	X		3
8								X	X	X	X	X	X	X	X		X	X					10
9	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X		21
10														X	X				X	X		X	5
11	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X					18
12																						X	1
14																						X	1
15																						X	1
18	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X					18
21																						X	1
22	X		X			X		X	X				X		X		X	X	X	X	X	X	13
23																			X	X	X	X	4
25																						X	1
26	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X					18
27	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X						17
28	X	X	X	X	X	X	X							X	X	X							10
30																						X	1
32																	X						1
33																						X	1
34																			X	X	X	X	4
35			X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X					16
38																						X	1
39	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X					18
40																	X						1
41																						X	1
42																						X	1
44	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X					18
45																						X	1
46									X														1
47								X	X	X	X	X	X										6
48																				X			1
49																						X	1
51			X			X	X	X	X			X											6
53																			X	X	X		3
54	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X					18
56																						X	1
57	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X					18
58																			X	X	X		3
60																	X	X	X	X			4
61	X	X						X	X	X	X	X	X	X	X	X	X	X				X	14
62																			X	X	X		3
63	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X					18
64	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X					18
65																			X	X	X	X	4
67			X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X					16
70	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X							16
71																			X	X	X		3
72																			X	X	X		3
74	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X		X	X					17
75	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X							16
76	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X								15
78																						X	1
79																						X	1
80	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X		X	X					17
82					X	X	X	X				X		X	X								7
83																						X	1
84	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X		X	X					17

Table A3. Continued.

Team	Pain	Medication	Energy	Mobility	Sleep	Activities	Work ability	Enjoy life	Meaningfulness	Concentration	Satisfaction appearance	Self esteem	Negative affect	Relationships	Social support	Sexual satisfaction	Quality life	Satisfaction health	Mean physical	Mean psychological	Mean social	Mean overall	Counts
86																			X	X	X	X	4
87																						X	1
88																			X	X	X	X	4
89	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X					18
91	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X					18
92																						X	1
103								X	X			X	X	X	X		X						7
105	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X					18
107	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X					18
108			X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X					16
109														X	X				X	X			4
110																	X						1
111																				X			1
113																						X	1
117																			X	X	X	X	4
118																			X	X	X		3
120	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X								15
123	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X		X						16
126																	X	X	X	X	X		5
129																						X	1
130																	X						1
132																						X	1
133			X	X	X	X	X	X	X	X	X	X		X	X	X	X	X					15
134																						X	1
136	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X					18
138	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X							16
141																			X	X	X		3
143																						X	1
146																						X	1
147								X	X			X	X				X	X					6
148																						X	1
149																						X	1
151	X	X	X	X	X	X	X	X	X	X	X	X		X	X		X	X					16
152																			X	X	X		3
155	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X					18
157																						X	1
158																			X	X	X	X	4
159																						X	1
162																				X			1
164																	X	X	X	X	X		5
165								X	X	X	X	X	X				X	X					8
168	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X							16
175																						X	1
176																	X	X					2
178	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X					18
179	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X		X	X					17
181																	X		X	X		X	4
183																						X	1
184	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X		X	X					17
185	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X					18
186																			X	X	X	X	4
187								X	X	X	X	X	X										6
188																						X	1
189																	X						1
192								X	X	X	X	X	X	X	X	X							9
193	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X					18
196																						X	1
	32%	32%	36%	34%	35%	37%	36%	43%	43%	38%	38%	42%	40%	40%	41%	28%	41%	33%	21%	24%	18%	38%

Table A4. Items included as dependent variables for research question 2 by each team.

Team	Pain	Medication	Energy	Mobility	Sleep	Activities	Work ability	Enjoy Life	Meaningfulness	Concentration	Satisfaction appearance	Self esteem	Negative affect	Relationships	Social support	Sexual satisfaction	Quality life	Satisfaction health	Mean physical	Mean psychological	Mean social	Mean overall	Counts
3	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X							16
5	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X					18
7																			X	X	X		3
8								X	X	X	X	X	X	X	X		X	X					10
9	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X		21
10														X	X				X	X		X	5
11	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X					18
12																						X	1
14																						X	1
15																						X	1
18	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X					18
21																						X	1
22	X		X			X		X	X				X		X		X	X	X	X	X	X	13
23																			X	X	X	X	4
25																						X	1
26	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X					18
27	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X						17
30																						X	1
32																	X						1
33																						X	1
34																			X	X	X	X	4
35			X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X					16
38	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X					18
39	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X					18
40																	X						1
41																						X	1
42																						X	1
44	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X					18
45																						X	1
46									X														1
47								X	X	X	X	X	X										6
48																				X			1
49																						X	1
51			X			X	X	X	X			X											6
53																			X	X	X		3
54	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X					18
56																						X	1
57	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X					18
58																			X	X	X		3
60																	X	X	X	X			4
61	X	X						X	X	X	X	X	X	X	X	X	X	X				X	14
62																						X	1
63	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X					18
64	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X					18
65																			X	X	X	X	4
67			X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X					16
70	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X							16
71																			X	X	X		3
72																			X	X	X		3
74	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X		X	X					17
75	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X							16
76	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X								15
78																						X	1
79																						X	1
80	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X		X	X					17
82					X	X	X	X				X		X	X								7
83																						X	1
84	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X		X	X					17
86																			X	X	X	X	4
87																						X	1
88																			X	X	X	X	4
89	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X					18
91	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X					18
92																						X	1
103								X	X			X	X	X	X		X						7
105	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X					18
107	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X					18
108			X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X					16
109														X	X				X	X			4
110																	X						1
111																				X			1
113																						X	1
117																			X	X	X	X	4

Table A4. Continued.

Team	Pain	Medication	Energy	Mobility	Sleep	Activities	Work ability	Enjoy Life	Meaning- fulness	Concent- ration	Satisfaction appearance	Self esteem	Negative affect	Relation- ships	Social support	Sexual satisfaction	Quality life	Satisfaction health	Mean physical	Mean psychological	Mean social	Mean overall	Counts
118																			X	X	X		3
120	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X								15
123	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X		X						16
126																	X	X	X	X	X		5
129																						X	1
130																	X						1
132																						X	1
133			X	X	X	X	X	X	X	X	X	X		X	X	X	X	X					15
134																						X	1
136	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X					18
138	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X							16
141																			X	X	X		3
143																						X	1
146																						X	1
147								X	X			X	X				X	X					6
148																						X	1
149																						X	1
151	X	X	X	X	X	X	X	X	X	X	X	X		X	X		X	X					16
152																			X	X	X		3
155	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X					18
157																						X	1
158																			X	X	X	X	4
159																						X	1
162																				X			1
164																	X	X	X	X	X		5
165								X	X	X	X	X	X				X	X					8
168	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X							16
175																						X	1
176																	X	X					2
178	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X					18
179	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X		X	X					17
181																	X		X	X		X	4
183																						X	1
184	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X		X	X					17
185	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X					18
186																			X	X	X	X	4
187								X	X	X	X	X	X										6
188																						X	1
189																	X						1
192								X	X	X	X	X	X	X	X	X							9
193	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X	X					18
196																						X	1
	33%	32%	37%	35%	36%	37%	37%	44%	44%	40%	40%	43%	41%	41%	42%	29%	43%	35%	21%	23%	17%	38%

Table A5. Items included as independent variables for research question 1 by each team.

Team	Service attendance	Prayer	Religious Status	Membership	Belief god	Belief afterlife	Spirituality	Norms lifestyle self	Norms god self	Norms lifestyle country	Norms god country	Counts
3	X	X	X	X	X	X	X	X	X			9
5	X	X	X	X	X	X	X	X	X			9
6	X											1
7	X	X								X		3
8	X	X			X	X	X	X	X	X	X	9
9	X	X	X	X	X	X	X	X	X			9
10	X	X	X	X	X	X	X	X	X			9
11	X	X	X	X	X	X	X	X	X			9
12	X	X	X	X	X	X	X	X	X			9
14	X	X	X	X	X	X	X	X	X			9
15	X	X	X	X	X	X	X	X	X			9
18			X									1
21	X	X			X	X						4
22	X	X	X	X	X	X	X	X	X			9
23	X	X	X	X	X	X	X	X	X			9
25	X	X	X		X	X	X	X	X			8
26	X	X	X	X	X	X	X	X	X			9
27			X		X	X	X		X			5
28	X	X			X	X	X	X	X			7
30			X									1
32								X	X			2
33			X									1
34	X	X	X	X	X	X	X	X	X			9
35					X	X	X	X	X			5
38	X	X	X	X	X	X	X	X	X			9
39	X	X			X	X						4
40			X									1
41			X	X	X	X	X	X	X			7
42	X	X			X	X	X	X	X			7
44	X	X	X	X	X	X	X	X	X			9
45	X	X	X	X	X	X	X	X	X			9
46	X	X	X	X	X	X	X	X	X			9
47	X	X	X		X	X	X	X	X			8
48	X	X			X	X	X	X	X			7
49			X									1
51	X	X	X	X	X	X	X	X	X			9
53	X	X	X		X	X	X	X	X			8
54	X	X	X		X	X	X	X	X			8
56	X	X						X	X			4
57	X	X	X	X	X	X	X	X	X			9
58			X									1
60	X	X	X	X	X	X	X	X	X			9
61	X	X	X	X	X	X	X	X	X			9
62	X	X	X	X	X	X	X	X	X			9
63	X	X	X	X	X	X	X	X	X			9
64	X	X	X	X	X	X	X	X	X			9
65	X	X			X	X	X	X	X			7
67	X	X			X	X	X	X	X			7
70	X	X			X	X	X	X	X			7
71								X	X			2
72	X	X	X	X	X	X	X	X	X			9
74	X	X			X			X	X			5
75	X	X	X		X	X	X	X	X			8
76	X	X			X	X	X	X	X			7
78	X	X			X	X	X	X	X			7
79			X									1
80	X	X	X	X	X	X	X	X	X			9
82	X	X			X							3
83	X	X	X	X	X	X	X	X	X			9
84	X	X	X	X	X	X	X	X	X			9
86	X	X	X	X	X	X		X	X			8
87	X	X	X	X	X	X	X	X	X			9
88	X	X	X		X	X	X	X	X			8
89	X	X			X	X	X	X	X			7
91	X	X			X	X	X	X	X			7
92	X	X	X	X	X	X		X	X			8
103	X		X				X	X				4
105	X	X	X	X	X	X	X	X	X			9
107	X	X	X	X	X	X	X	X	X			9
108			X	X		X		X	X			5
109	X	X	X	X	X	X	X	X	X			9
110			X									1
111								X	X			2
113												0
117	X	X	X		X	X		X	X			7
118	X	X			X	X	X					5
120	X	X			X	X		X				5
123	X	X	X	X	X	X	X	X	X			9
126			X									1

Table A5. Continued.

Team	Service attendance	Prayer	Religious Status	Membership	Belief god	Belief afterlife	Spirituality	Norms lifestyle self	Norms god self	Norms lifestyle country	Norms god country	Counts
129	X	X	X	X	X	X	X	X	X			9
130			X									1
132	X	X	X	X	X	X		X	X			8
133	X	X	X	X	X	X	X	X	X			9
134	X	X			X	X	X	X	X			7
136	X	X	X	X	X	X	X					7
138	X	X	X	X	X	X	X	X	X			9
141			X									1
143	X	X	X	X	X	X	X	X	X			9
146	X	X			X	X	X	X	X			7
147					X				X			2
148	X	X	X	X	X	X	X	X	X			9
149	X			X								2
151	X	X	X	X	X	X	X					7
152	X	X	X	X	X	X	X	X	X			9
155	X	X	X	X	X	X	X	X	X			9
157	X	X	X	X	X	X	X	X	X			9
158	X	X	X	X	X	X	X	X	X			9
159								X	X			2
162	X	X	X		X	X	X	X	X			8
164			X									1
165	X	X	X		X	X	X					6
168	X	X			X	X	X					5
175	X	X	X	X	X	X	X	X	X			9
176		X						X	X			3
178	X	X	X	X	X	X	X	X	X			9
179	X	X	X	X	X	X	X	X	X			9
181	X	X	X	X	X	X	X	X	X			9
183	X	X	X	X	X	X	X	X	X			9
184	X	X			X	X	X	X	X			7
185	X	X			X	X	X	X	X			7
186	X	X	X	X	X	X	X	X	X			9
187	X	X			X							3
188	X	X	X	X	X	X	X	X	X			9
189			X									1
192	X	X	X	X	X	X	X	X	X			9
193	X	X	X	X	X	X	X	X	X			9
196	X	X	X	X	X	X	X	X	X			9
	79%	78%	70%	50%	79%	76%	70%	76%	76%	2%	1%

Table A6. Items included as independent variables for research question 2 by each team.

Team	Service attendance	Prayer	Religious status	Membership	Belief god	Belief afterlife	Spirituality	Norms lifestyle self	Norms god self	Norms lifestyle country	Norms god country	External norms	Counts
3	X	X	X	X	X	X	X	X	X	X	X		11
5	X	X	X	X	X	X	X	X	X	X	X		11
6													0
7	X		X	X									3
8	X	X	X	X			X	X	X	X	X		9
9	X	X	X	X	X	X	X	X	X	X	X		11
10	X	X	X	X	X	X	X	X	X	X	X		11
11	X	X	X	X	X	X	X	X	X	X	X		11
12	X	X	X	X	X	X	X	X	X	X	X		11
14	X	X	X	X	X	X	X	X	X	X	X		11
15	X	X	X	X	X	X	X	X	X	X	X		11
18	X				X								2
21	X	X	X	X			X	X					6
22	X	X	X	X	X	X	X	X	X	X	X		11
23	X	X	X	X	X	X	X	X	X	X	X		11
25	X	X	X	X	X		X	X	X	X	X		10
26	X	X	X	X	X	X	X	X	X	X	X		11
27	X	X			X		X	X	X		X		7
28													0
30	X	X								X	X		4
32	X	X								X	X		4
33	X	X			X								3
34	X	X	X	X	X	X	X	X	X	X	X		11
35	X	X					X	X	X	X	X		7
38	X	X	X	X	X	X	X	X	X	X	X		11
39	X	X	X	X			X	X					6
40	X	X			X								3
41	X	X			X	X	X	X	X	X	X		9
42	X	X	X	X			X	X	X	X	X		9
44	X	X	X	X	X	X	X	X	X	X	X		11
45	X	X	X	X	X	X	X	X	X	X	X		11
46	X	X	X	X	X	X	X	X	X	X	X		11
47	X	X	X	X	X		X	X	X	X	X		10
48	X	X	X	X			X	X	X	X	X		9
49	X	X			X								3
51	X	X	X	X	X	X	X	X	X	X	X		11
53	X	X	X	X	X		X	X	X	X	X		10
54			X	X	X	X	X	X	X	X	X		9
56	X	X	X	X						X	X		6
57	X	X	X	X	X	X	X	X	X	X	X		11
58	X	X	X	X		X	X	X	X	X	X		10
60	X	X	X	X	X	X	X	X	X	X	X		11
61	X	X	X	X	X	X	X	X	X	X	X		11
62	X	X	X	X	X	X	X	X	X	X	X		11
63	X	X	X	X	X	X	X	X	X	X	X		11
64	X	X	X	X	X	X	X	X	X	X	X		11
65	X	X	X	X			X	X	X	X	X		9
67	X	X	X	X			X	X	X	X	X		9
70	X	X	X	X			X	X	X	X	X		9
71	X	X								X	X		4
72	X	X	X	X	X	X	X	X	X	X	X		11
74	X	X	X	X			X			X	X		7
75	X	X	X	X	X		X	X	X	X	X		10
76	X	X	X	X			X	X	X	X	X		9
78	X	X	X	X			X	X	X	X	X		9
79	X	X			X								3
80	X	X	X	X	X	X	X	X	X	X	X		11
82	X	X	X	X			X						5
83	X	X	X	X	X	X	X	X	X	X	X		11
84	X	X	X	X	X	X	X	X	X	X	X		11
86	X	X	X	X	X	X	X	X		X	X		10
87	X	X	X	X	X	X	X	X	X	X	X		11
88	X	X	X	X	X		X	X	X	X	X		10
89	X	X	X	X			X	X	X	X	X		9
91	X	X	X	X			X	X	X	X	X		9
92	X	X	X	X	X	X	X	X		X	X		10
103	X		X		X				X	X			5
105	X	X	X	X	X	X	X	X	X	X	X		11
107	X	X	X	X	X	X	X	X	X	X	X		11
108	X	X			X	X		X		X	X		7
109	X	X	X	X	X	X	X	X	X	X	X		11
110	X				X								2
111	X	X								X	X		4
113													0
117	X	X	X	X	X		X	X		X	X		9
118	X	X	X	X			X	X	X				7
120	X	X	X	X			X	X		X			7
123	X	X	X	X	X	X	X	X	X	X	X		11

Table A6. Continued.

Team	Service attendance	Prayer	Religious status	Membership	Belief god	Belief afterlife	Spirituality	Norms lifestyle self	Norms god self	Norms lifestyle country	Norms god country	External norms	Counts
126	X				X								2
129	X	X	X	X	X	X	X	X	X	X	X		11
130	X	X			X								3
132	X	X	X	X	X	X	X	X		X	X		10
133	X	X	X	X	X	X	X	X	X	X	X		11
134	X	X	X	X			X	X	X	X	X		9
136	X	X	X	X	X	X	X	X	X				9
138			X	X	X	X	X	X	X	X	X	X	10
141	X	X	X	X	X	X	X	X	X	X	X		11
143	X	X	X	X	X	X	X	X	X	X	X		11
146	X	X	X	X			X	X	X	X	X		9
147			X	X						X			3
148	X	X	X	X	X	X	X	X	X	X	X		11
149	X	X	X			X							4
151	X	X	X	X	X	X	X	X	X				9
152	X	X	X	X	X	X	X	X	X	X	X		11
155	X	X	X	X	X	X	X	X	X	X	X		11
157	X	X	X	X	X	X	X	X	X	X	X		11
158	X	X	X	X	X	X	X	X	X	X	X		11
159	X	X								X	X		4
162	X	X	X	X	X		X	X	X	X	X		10
164	X	X			X								3
165			X	X	X		X	X	X				6
168	X	X	X	X			X	X	X				7
175	X	X	X	X	X	X	X	X	X	X	X		11
176	X	X		X						X	X		5
178	X	X	X	X	X	X	X	X	X	X	X		11
179	X	X	X	X	X	X	X	X	X	X	X		11
181	X	X	X	X	X	X	X	X	X	X	X		11
183	X	X	X	X	X	X	X	X	X	X	X		11
184			X	X			X	X	X	X	X		7
185	X	X	X	X			X	X	X	X	X		9
186	X	X	X	X	X	X	X	X	X	X	X		11
187	X	X	X	X			X						5
188	X	X	X	X	X	X	X	X	X	X	X		11
189	X	X			X								3
192	X	X	X	X	X	X	X	X	X	X	X		11
193	X	X	X	X	X	X	X	X	X	X	X		11
196	X	X	X	X	X	X	X	X	X	X	X		11
	93%	89%	80%	79%	68%	53%	79%	77%	71%	79%	77%	1%

Table A7. Covariates included for research question 1 by each team.

Team	None	Age	Country	Denomination	GDP Scaled	Gender	SES	Attention check	Education	Ethnicity	Norms lifestyle country	Norms god country	Compensation	Sample type	GDP	Service attendance	Prayer	Religious status	Membership	Belief God	Belief afterlife	Spirituality	Norms lifestyle self	Norms god self	External norms	External N subjects	Counts
3	X																										1
5		X	X	X	X	X																					5
6		X				X																					2
7		X				X	X																				3
8		X				X	X	X	X																		5
9		X				X			X																		3
10					X	X			X																		3
11	X																										1
12	X																										1
14	X																										1
15		X			X	X	X		X																		5
18		X				X	X		X																		4
21		X			X	X	X		X																		5
22	X																										1
23		X					X		X																		3
25		X				X	X		X																		4
26		X			X	X	X		X																		5
27	X																										1
28		X		X		X			X	X																	5
30	X																										1
32		X	X	X		X			X		X	X	X	X													9
33							X																				1
34											X	X															2
35							X								X												2
38	X																										1
39		X				X	X		X																		4
40											X	X															2
41							X		X																		2
42		X		X		X	X		X	X					X												7
44		X				X	X		X						X												5
45		X			X	X	X		X		X	X															7
46		X				X	X		X																		4
47	X																										1
48											X	X															2
49		X	X																								2
51							X																				1
53		X			X	X	X		X																		5
54		X		X	X	X	X		X	X																	7
56											X	X															2
57		X				X	X		X																		4
58	X																										1
60	X																										1
61		X		X		X	X		X	X																	6
62	X																										1
63		X			X	X	X		X	X	X	X		X													9
64	X																										1
65		X				X	X								X												4
67		X				X	X		X	X																	5
70		X				X	X		X						X												5
71	X																										1
72	X																										1
74							X																				1
75		X			X	X			X																		4
76											X	X															2
78		X					X																				2
79	X																										1
80		X					X		X																		3
82	X																										1
83		X	X			X	X		X																		5
84		X	X			X	X		X					X													6
86		X					X		X																		3
87	X																										1
88		X					X		X																		3
89	X																										1
91		X			X	X	X							X													5
92	X																										1
103		X					X																				2
105		X	X	X	X	X	X		X	X	X	X	X	X													12
107	X																										1
108		X				X								X													3
109		X				X																					2
110	X																										1
111	X																										1
113		X			X	X	X		X		X	X	X	X		X	X	X	X	X	X	X	X	X			18
117			X			X	X		X																		4
118	X																										1

Table A7. Continued.

Team	None	Age	Country	Denomination	GDP Scaled	Gender	SES	Attention check	Education	Ethnicity	Norms lifestyle country	Norms god country	Compensation	Sample type	GDP	Service attendance	Prayer	Religious status	Membership	Belief God	Belief afterlife	Spirituality	Norms lifestyle self	Norms god self	External norms	External N subjects	Counts
120		X					X		X																		3
123		X			X	X	X		X																		5
126		X				X			X																		3
129		X	X				X		X																		4
130	X																										1
132		X				X	X		X																		4
133		X									X	X			X												4
134		X	X			X	X		X																		5
136																							X	X			2
138		X			X	X	X		X		X	X													X		8
141	X																										1
143		X		X		X	X			X			X	X	X											X	9
146						X			X																		2
147		X				X	X		X																		4
148		X		X	X	X	X		X																		6
149		X			X	X	X																				4
151		X				X	X		X	X	X	X	X	X	X												10
152		X			X	X	X																				4
155		X					X																				2
157		X		X		X	X		X		X	X															7
158		X				X	X		X																		4
159		X				X	X																				3
162		X		X	X	X	X		X																		6
164		X				X			X						X												4
165		X				X	X		X		X	X															6
168		X				X	X				X	X															5
175	X																										1
176		X				X	X																				3
178	X																										1
179		X				X	X		X						X												5
181		X			X	X	X		X	X				X													7
183	X																										1
184									X																		1
185		X		X	X	X	X		X	X																	7
186			X				X		X	X																	4
187			X								X	X															3
188		X	X	X		X	X		X	X			X	X													9
189		X				X			X																		3
192		X				X	X		X					X	X												6
193			X			X	X		X		X	X			X												7
196		X		X		X	X		X					X													6
	25%	60%	11%	12%	18%	54%	56%	1%	50%	11%	15%	15%	5%	11%	10%	1%	1%	1%	1%	1%	1%	1%	2%	2%	1%	1%

Table A8. Covariates included for research question 2 by each team.

Team	None	Age	Country	Denomination	GDP scaled	Gender	SES	Attention check	Education	Ethnicity	Norms lifestyle country	Norms god country	Compensation	Sample type	GDP	Service attendance	Prayer	Religious status	Membership	Belief God	Belief afterlife	Spirituality	Norms lifestyle self	Norms god self	External N subjects	External religious majority	Counts
3	X																										1
5		X	X	X	X	X																					5
6																											0
7		X				X	X																				3
8		X				X	X	X	X																		5
9		X				X			X																		3
10					X	X			X																		3
11	X																										1
12	X																										1
14	X																										1
15		X			X	X	X		X																		5
18		X				X	X		X																		4
21		X			X	X	X		X																		5
22	X																										1
23							X																				1
25		X				X	X		X																		4
26		X			X	X	X		X																		5
27	X																										1
28																											0
30		X			X		X																				3
32		X	X	X		X			X	X	X																7
33							X																				1
34	X																										1
35							X					X															2
38	X																										1
39		X				X	X		X																		4
40	X																										1
41							X		X																		2
42																											0
44		X				X	X		X			X															5
45		X			X	X	X		X																		5
46		X				X	X		X																		4
47	X																										1
48	X																										1
49		X	X																								2
51							X																				1
53		X			X	X	X		X																		5
54		X		X	X	X	X		X				X														7
56	X																										1
57		X				X	X		X																		4
58		X					X		X			X															4
60	X																										1
61		X		X		X	X		X				X														6
62	X																										1
63		X			X	X	X		X		X		X														7
64	X																										1
65		X				X	X					X															4
67		X				X	X		X				X														5
70		X				X	X		X			X															5
71	X																										1
72	X																										1
74							X																				1
75		X			X	X			X																		4
76	X																										1
78		X					X																				2
79	X																										1
80		X					X		X																		3
82	X																										1
83		X	X			X	X		X																		5
84		X	X			X	X		X		X																6
86		X					X		X																		3
87	X																										1
88		X					X		X																		3
89	X																										1
91		X			X	X	X				X																5
92	X																										1
103		X					X																				2
105		X	X	X		X	X		X	X	X	X	X														10
107	X																										1
108		X				X					X																3
109		X																									1
110	X																										1
111	X																										1
113		X			X	X	X		X	X	X			X	X	X	X	X	X	X	X	X	X	X			18
117			X			X	X		X																		4
118	X																										1

Table A8. Continued.

Team	None	Age	Country	Denomination	GDP scaled	Gender	SES	Attention check	Education	Ethnicity	Norms lifestyle country	Norms god country	Compensation	Sample type	GDP	Service attendance	Prayer	Religious status	Membership	Belief God	Belief afterlife	Spirituality	Norms lifestyle self	Norms god self	External N subjects	External religious majority	Counts
120		X					X		X																		3
123		X			X	X	X		X																		5
126		X				X			X																		3
129		X	X				X		X																		4
130	X																										1
132		X				X	X		X																		4
133		X										X															2
134		X	X			X	X		X																		5
136																							X	X			2
138		X			X	X	X		X																		5
141	X																										1
143		X		X		X	X			X	X	X	X												X		9
146						X			X																		2
147		X				X	X		X																		4
148		X		X	X	X	X		X																		6
149		X			X	X	X																				4
151		X				X	X		X	X	X	X	X														8
152		X			X	X	X																				4
155		X					X																				2
157		X		X		X	X		X																		5
158		X				X	X		X																		4
159		X				X	X																				3
162		X		X	X	X	X		X																		6
164		X				X			X			X															4
165		X				X	X		X																		4
168		X				X	X																				3
175	X																										1
176		X				X	X																				3
178	X																										1
179		X				X	X		X			X															5
181		X			X	X	X		X		X		X														7
183	X																										1
184							X		X																	X	3
185		X		X	X	X	X		X				X														7
186			X				X		X				X														4
187			X																								1
188		X	X	X		X	X		X	X	X		X														9
189		X				X			X																		3
192		X				X	X		X		X	X															6
193			X			X	X		X			X															5
196		X		X		X	X		X		X																6
	27%	58%	11%	10%	18%	50%	57%	1%	49%	5%	11%	11%	9%	1%	1%	1%	1%	1%	1%	1%	1%	1%	2%	2%	1%	1%