How Political Efficacy Relates to Online and Offline Political Participation: A Multilevel Meta-analysis

ABSTRACT

The rapid rise of digital media use for political participation has coincided with an increase in concerns about citizens’ sense of their capacity to impact political processes. These dual trends raise the important question of how people’s online political participation is connected to perceptions of their own capacity to participate in and influence politics. The current study overcomes the limitation of scarce high-quality cross-national and over-time data on these topics by conducting a meta-analysis of all extant studies that analyze how political efficacy relates to both online and offline political participation using data sources in which all variables were measured simultaneously. We identified and coded 48 relevant studies (with 184 effects) representing 51,860 respondents from 28 countries based on surveys conducted between 2000 and 2016. We conducted a multilevel random effects meta-analysis to test the main hypothesis of whether political efficacy has a weaker relationship with online political participation than offline political participation. The findings show positive relationships between efficacy and both forms of participation, with no distinction in the magnitude of the two associations. In addition, we tested hypotheses about the expected variation across time and democratic contexts, and the results suggest contextual variation for offline participation but cross-national stability for online participation. The findings provide the most comprehensive evidence to date that online participation is as highly associated with political efficacy as offline participation, and that the strength of this association for online political participation is stable over time and across diverse country contexts.

KEYWORDS:

• Political efficacy
• online political participation
• offline political participation
• multilevel random effects meta-analysis

The use of digital media for political participation has increased steadily during the past twenty years (Boulianne, 2020). This same time period has also been characterized by concerns about changing relationships between citizens and the state, including the phenomena of democratic backsliding and decreased democratic legitimacy (Kriesi, 2013, 2020; Lührmann & Lindberg, 2019; Waldner & Lust, 2018). These contemporaneous trends raise the question of whether people perceive that their online political actions are effective ways to engage in political processes.

Astute observers of recent trends in global political participation such as Erica Chenoweth (2020) have clearly articulated both the promise and potential pitfalls of online and digital political participation. Discussing the challenge of contemporary movements’ tactical reliance on online political participation, Chenoweth (2020, p. 78) notes the advantages of swift recruitment and the capacity to communicate grievances to broad audiences, as well as the disadvantages of diminished capacity to recruit people into effective organizations that are able to strategize to achieve collective goals. Related scholarship on “slacktivism” or “clicktivism,” which Freelon et al. (2020, p. 1197) described as prominent in early online participation literature, was concerned that the low-cost and symbolic nature of activities such as posting on social media may serve to “project an impression of efficacy without actually being effective.”

Early research on online forms of participation noted by Freelon et al. (2020, p. 1197) focused on two main concerns: first, that this type of activity may not be politically consequential; and second, that it may also act as a substitute for more effective and impactful types of participation. While empirical evidence has repeatedly shown that digital activities do not replace offline activities and instead complement them (e.g., Boulianne & Theocharis, 2020; Gil de Zúñiga et al., 2017; Lane et al., 2017), little is known regarding whether political participants themselves consider their online political activities to be (in)consequential or (in)effective in comparison to their offline engagement. This question is increasingly important as the range and prevalence of online political activities have expanded over time, including activities such as signing petitions online, contacting officials through social media, and expressing opinions online (Boulianne, 2020; Theocharis, 2015; Theocharis & de Moor, 2021; Theocharis & van Deth, 2018). Further, research shows high levels of online political engagement among young people (Boulianne & Theocharis, 2020), suggesting a generational shift in political communication patterns. These trends highlight the importance of assessing whether political participants themselves believe that their online political actions are meaningful.

The current study investigates this topic by assessing whether the relationship between individuals’ perceptions of their political efficacy (PE) and their political participation is weaker for online political participation (OnPP) than offline political participation (OffPP). We test the main hypothesis (H1) regarding efficacy by conducting a multilevel random effects meta-analysis using the most comprehensive evidence available. In addition, we investigate two hypotheses about the expected variation in the relationships between PE, OnPP and OffPP: over time (H2) and across varying national contexts (H3). The concluding discussion reviews the implications of the findings as digital activism continues to expand, and outlines additional avenues for investigating the causal relations between political efficacy and different types of political participation in the digital era.

Political Participation and Political Efficacy

Do people perceive that their political actions make a difference? This question is particularly important in an era marked by concerns about both democratic legitimacy (Dahlberg et al., 2015; Kriesi, 2014, 2020) and the potentially negative effects of online activity on democratic society (Anduiza et al., 2019; Vraga, 2019). Studies investigating the OnPP-OffPP distinction in terms of sociodemographic correlates using high-quality representative survey data (e.g., Gibson & Cantijoch, 2013; Oser et al., 2013; Schradie, 2018; Vaccari, 2017) have found that online participation has a stronger association with socio-economic status compared to offline participation. Consensus has not yet emerged in the literature, however, regarding the question of how people’s self-perceived efficacy relates to their involvement in online versus offline political participation.

In efforts to understand the connection between political participation and democratic governance, political efficacy has been viewed as a key attitudinal measure, dating back to Campbell et al.’s (1954) study of how voters make decisions. The classic definition of political efficacy articulated by Campbell et al. (1954, p. 187) – “the feeling that individual political action does have, or can have, an impact upon the political process” – is still widely cited in contemporary scholarship. By the early 1990s, empirical research clarified this conceptual distinction based on studies drawing on the American National Election Studies (ANES). Niemi et al.’s (1991, pp. 84–85) description of these concepts has informed subsequent research on internal efficacy, defined as “beliefs about one’s own competence to understand, and to participate effectively in, politics,” and external efficacy, defined as “beliefs about the responsiveness of governmental authorities and institutions to citizen demand.” Decades of research on offline participation has shown that both types of political efficacy tend to have a positive association with offline political participation, with internal efficacy having a stronger association with OffPP than external efficacy (e.g., Chamberlain, 2012; Morrell, 2003).

In contrast to the relatively limited available survey data on the PE-OnPP connection, extensive high-quality cross-sectional and longitudinal data on the PE-OffPP connection have been analyzed in numerous prominent studies that use data from the American National Election Studies (e.g., Niemi et al., 1991; Robison et al., 2018), the International Social Survey Programme (e.g., Vráblíková, 2014), and the Comparative Study of Electoral Systems (e.g., Karp & Banducci, 2008). The challenge of relatively limited large-scale comparative survey data about online phenomena was articulated in Theocharis’s 2015 observation (p. 2) that, at the time, prominent surveys such as the European Social Survey, the European Values Study, the European Election Study, and the World Values Survey did not include any questions on these topics. This important task was therefore left to country-specific academic projects. While select questions about online political participation have been added to the survey instruments of some of the large-scale comparative surveys in recent years, the lack of longitudinal data from these sources precludes analysis of longer term trends.

Thus, due to the limited availability of high-quality survey data on online political participation, most of the research on the PE-OnPP relationship is based on country-specific cross-sectional studies. A series of country-specific studies have produced robust empirical evidence on other related topics, such as showing that online political participation is effective at mobilizing people to be politically active offline as well (Bode, 2017; Boulianne & Theocharis, 2020; Cantijoch et al., 2016; Kahne & Bowyer, 2018; Karpf, 2010; Kwak et al., 2018). Yet, the question remains open as to whether the PE-OnPP association has a positive strength similar to that of the PE-OffPP association, as country-specific studies that analyze data from varied contexts and time periods have thus far yielded conflicting results.

For example, consistent with research that conceptualizes political efficacy as more of an inherent “trait” than a context-dependent “state” (Schneider et al., 2014), some digital media research indicates similarly large positive associations between political efficacy and both types of participation (online and offline). This pattern of a strong positive PE-OnPP and PE-OffPP connection is evident in studies such as Jung et al.’s (2011) analysis of data from a 2008 survey weighted to be representative of the national US population, and Park’s (2015) representative survey in South Korea conducted in 2012. Consistent with the theoretical rationale offered by social-cognitive theory (e.g., Bandura, 2001), these findings indicate that new media affordances may serve to provide those who are politically interested and efficacious with opportunities to engage in politics, regardless of the online or offline political context. In contrast, some studies report negative associations for PE-OnPP along with a positive connection for PE-OffPP, indicating a weaker relationship between efficacy and political participation in online contexts. This pattern of a weaker connection between PE-OnPP than PE-OffPP is reported in studies such as Stromer-Galley’s (2002) analysis of a 2000 survey of US citizens with and without Internet access, and Zhu et al.’s (2017) survey of students in Hong Kong. In addition to emphasizing the need to synthesize conflicting extant findings to assess the generalizable relationship between PE and OnPP/OffPP, recent scholarship on contextualizing evidence in political communication research (Esser, 2019; Rojas & Valenzuela, 2019) clarifies the importance of investigating whether these relationships change in systematic ways over time and across country contexts.

To synthesize these heterogeneous findings on the relationship between political efficacy and different types of political participation, we focus on OnPP and OffPP in the context of both classic and more recent literature on the expression of the will of the people as a crucial component of democratic governance (Bennett, 2012; Bennett & Segerberg, 2012; Dahl, 1961; Mill, [1861] 1962). Most research on the connection between political behavior and representation has focused on electoral-oriented participation (Dassonneville et al., 2021; Powell, 2004). Yet some researchers have proposed that the underlying mechanism that links voting and electoral-oriented participation to responsiveness is that individuals who vote are also more likely to participate in additional ways that influence public opinion and decision-makers (Bartels, 2016; Giugni & Grasso, 2019a; Griffin & Newman, 2005). Although the logic of this communication mechanism in the literature is also relevant for online political participation (Bennett, 2012; Bennett & Segerberg, 2012), empirical studies that have investigated how participation beyond voting may enhance representation have focused primarily on offline political participation behaviors such as civic activism and protest activity (Htun & Weldon, 2012; Leighley & Oser, 2018; Rasmussen & Reher, 2019).

A related line of research that anticipates empirical support for a weaker association between PE-OnPP compared to PE-OffPP has focused on the degree to which online political activities described as “clicktivism” or “slacktivism” may constitute token displays of political support (Chou et al., 2020; Kristofferson et al., 2014). Both normative and empirical studies have discussed concerns that this type of activity both requires little effort and has minimal impact (Christensen, 2011; Freelon et al., 2020; Halupka, 2014; Karpf, 2012), and may even be primarily focused on entertainment (Theocharis & Lowe, 2016; Theocharis & Quintelier, 2016). An even more concerning possibility discussed in the scholarly literature and public discourse is that online participation may serve as a channel to express frustration with no expectation of political change, and thus can potentially demobilize citizens who would otherwise have been politically engaged (Anduiza et al., 2012; Gladwell, 2010).

Beyond discussions of clicktivism and slacktivism, the literature includes two main arguments for why political efficacy may have a weaker relationship with OnPP than OffPP. First, OffPP is expected to require more intentional effort and investment than OnPP (Boulianne, 2020; Karpf, 2010; Schlozman et al., 2010). The potentially smaller effort and investment required to participate online versus offline may impact participants’ sense of their own political efficacy, prompting a perception that OnPP is a less meaningful form of participation than OnPP. Second, participants may perceive a clearer connection between offline political participation and political outcomes, as political institutions and processes are often considered to be better designed to respond to offline and more traditional modes of participation (Christensen, 2011; Matthews, 2021). These arguments relate to the expected role of online versus offline participation in mobilizing citizens into political action and facilitating their sense of efficacy through a meaningful connection between political participation and the achievement of political representation.

While innovative research designs aimed at studying the contemporary mechanisms linking online participation and representational outcomes are just beginning to emerge (Blumenau, 2021; Ennser-Jedenastik et al., 2022; Matthews, 2021), conducting research on longer-term structural trends in the relationship between online participation and individual attitudes is more challenging due to the lack of high-quality long-term comparative data on online participation (Theocharis, 2015; Vaccari & Valeriani, 2018). In light of these data constraints, an important avenue for investigating the relationship between online political participation and representation is to synthesize extant studies on this topic, which are mostly cross-sectional, single-country studies, in order to assess the relation between citizens’ sense of their own political efficacy and the types of participation in which they are active.

If participants themselves view online participation as ineffective, or even akin to a frustration valve, then empirical evidence would show a low or perhaps even negative association between this type of political behavior and individuals’ sense of their capacity to engage in and affect political processes and outcomes. Thus, the current study investigates the relationship between political efficacy and two main categories of political behavior: online and offline political participation. The classic definition of the distinction between these two types of behaviors as described in the literature is that online participation is activity that takes place through digital means, whereas offline participation is conducted as more traditional in-person activities that involve physical presence (Gibson & Cantijoch, 2013). We follow the common approach of investigating differences between offline and online participation (Matthes et al., 2019; Strömbäck et al., 2018), while acknowledging the theoretical and empirical challenge discussed in recent literature of establishing definitive long-term categories of different types of political participation (Ohme et al., 2018; Ruess et al., Forthcoming; Theocharis, 2015; Theocharis & van Deth, 2018). The research design therefore allows an empirical investigation of whether there are differences in the strength of the association between political efficacy and these two types of participation over time.

Research Question and Hypotheses¹

Building on this literature, the current study investigates the main research question of how political efficacy relates to OnPP compared to OffPP. The main hypothesis of this study is that individuals’ perceptions of their political efficacy are more weakly related to OnPP than to OffPP. Specifically, the first hypothesis is as follows: Political efficacy is less strongly connected to online than to offline participation (H1: Efficacy).

In addition to this main hypothesis, the literature suggests the importance of testing for two moderating effects on the relationship between political efficacy and OnPP versus OffPP. An important line of research indicates that these two types of behaviors have become less distinct over time, as online activity and digital media have become an ongoing feature of daily life (Farrell, 2012; Ohme et al., 2018). The increased integration of digital activities as essential aspects of everyday life would mean that any difference in the relationship between political efficacy and OnPP versus OffPP identified in the early years of the Internet may have decreased over time. The second hypothesis is based on this over-time expectation: The relationship between PE and OnPP and the relationship between PE and OffPP have converged in recent years (H2: Over-time convergence).

Finally, drawing on the global reach of the focal studies, we test the expectation that political efficacy and OnPP are more strongly related in established democratic contexts characterized by institutions and representatives that favor responsiveness to the will of the public (Giugni & Grasso, 2019a; Wagner et al., 2017). The rationale behind this hypothesis emerges from literature on democratic governance and responsiveness. Research on the effect of political participation on representational outcomes discusses a communication mechanism that may enhance the representation of those who participate beyond the electoral arena, particularly in strong democratic contexts (Bartels, 2016; Giugni & Grasso, 2019b; Griffin & Newman, 2005). Therefore, the third hypothesis states: The relationship between PE and OnPP is stronger in stronger electoral democracies (H3: Democratic context).

Data and Methods²

Study Search and Selection Procedures

For research questions and hypotheses about which there is a diverse set of findings in the political communications literature, meta-analysis is an increasingly common approach for synthesizing research results and establishing definitive findings (e.g., Boulianne, 2020; Matthes et al., 2018; Munzert & Ramirez-Ruiz, 2021; Walter et al., 2020; Zoizner, 2021). To identify relevant studies, we conducted a series of systematic search and selection procedures between March 2019 and December 2020 that we documented according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines (Liberati et al., 2009; see Supplementary Material for PRISMA figure).

The main selection criterion used to identify relevant studies was that all three relevant variables (PE, OnPP and OffPP) were collected in the same survey sample and analyzed to assess the association between political efficacy with the separate dependent variables of the two key forms of political participation (OnPP and OffPP). We first searched the academic database Scopus to identify studies that met the key search criteria.³ We followed the standard search practice of reviewing the works cited in systematic reviews of related topics (Lutz et al., 2014; Robison et al., 2018; Ruess et al., Forthcoming), scanning the reference lists of the studies that met the selection criteria to identify additional relevant studies, and reviewing additional research conducted by authors of the studies that met the inclusion criteria. Further, we presented preliminary findings of the study in a series of conference and seminar talks between March and September of 2020, during which we asked content experts to identify any additional studies that met the search criteria.⁴ Finally, we emailed the authors of the studies that met the inclusion criteria to inquire whether they had authored or were aware of any additional unpublished studies that met the inclusion criteria.

These search procedures identified 1,356 studies for potential inclusion. A first set of selection steps excluded duplicate studies, studies that lacked relevant quantitative analysis of survey data, or were in a language other than English. The remaining selection steps focused on retaining studies that properly measured the key constructs, and thus excluded indices of political participation that lacked a clear distinction between offline and online participation; limited the sample to studies that used self-reported measures of actual political behavior (rather than likelihood of participating); and omitted effects from models that included an interaction effect but did not also report on the constituent variables’ main effects. For unpublished studies, we conducted a search to determine whether a more recent version of the study had been published, and we included unpublished studies if we did not identify a published version and there were no clear flaws in the study.

Data Coding and Analysis

This search and selection process yielded a final set of 48 studies that we coded according to a coding protocol designed to test our hypotheses, and then conducted intercoder checks of the coded data (see Supplemental Material for the coding protocol). The full dataset of 184 coded effects represents a total sample of n = 51,860 respondents from 28 countries. As expected, due to the lack of high-quality cross-national surveys on these topics by large-scale survey programs, most of the studies that met our selection criteria analyze country-specific cross-national datasets. Since a select number of these studies analyze the same or similar data samples, we assigned the code of a unique dataset number to each of the 41 distinctive datasets.

The main coding distinction relevant for the analysis of political participation is between online and offline political participation. For political efficacy, we included studies that analyzed all forms of political efficacy measures, and we coded the type of political efficacy (internal, external, general, and other) used in each study, following established research on political efficacy types (Kenski & Stroud, 2006; Morrell, 2003; Niemi et al., 1991). We follow the classic definitional distinctions in the political efficacy literature (e.g., Morrell, 2003; Niemi et al., 1991) for coding internal efficacy as measures that focus on individuals’ beliefs about their own capacity to understand and engage in politics; external efficacy as beliefs about the political responsiveness to citizens; and general efficacy as all measures that reference the broad concept of political efficacy with no internal/external distinction. In addition, a number of specific political efficacy measures have emerged in the more recent literature that are not fully captured by these categories (Pingree et al., 2014). These measures are coded in our data as other efficacy, including measures of “collective efficacy,” “self-efficacy,” and “internet efficacy” (see Supplementary Material for documentation of the survey questions used for these “other efficacy” measures).

As noted, the theoretical focus of our contribution is to assess whether political efficacy is more weakly associated with OnPP than OffPP. Thus, the target quantity that we aim to analyze is the magnitude of the association between PE-OnPP in comparison to PE-OffPP. Informed by recent research on causal interpretation and regression models (e.g., Keele et al., 2020), our theoretical focus clarifies that the analytical goal of the current study is not causal inference, but rather to conduct a rigorous estimate of the magnitude of the association between political efficacy and these two forms of participation. A long-standing analytical approach in meta-analytic research is to code and analyze the beta coefficients of control variables in multivariate models, motivated in part by this approach’s facilitation of meta-analyses which contribute to cumulative research (see, Hünermund and Louw [2020, p. 6] for a review of this argument). Yet, recent research highlights the problem of biased estimates, and emphasizes the importance of clearly identifying and analyzing the study’s intended estimand, i.e., the target quantity that the study aims to assess and analyze (Hünermund & Louw, 2020; Lundberg et al., 2021).

With analytical focus on both contributing to cumulative research and analyzing our intended estimand as clearly as possible, we use two meta-analytic datasets to test each of our hypotheses. First, following standard practice in contemporary high-impact meta-analysis to transform different types of effect estimates from multivariate analyses into a common metric (see for example, Amsalem & Zoizner, 2022; Dinesen et al., 2020; Matthes et al., 2019; Walter et al., 2020), we converted the k = 184 coded effects from multivariate models into k = 183 common effect estimates.⁵ These common effect estimates are converted from three main types of coded effects from multivariate models: standardized regression coefficients, unstandardized regression coefficients, and odds ratios (see Supplementary Material for a summary of the number of samples and effects derived from distinct types of regression models). We used Peterson and Brown’s (2005) approach to convert the standardized coefficients to Pearson’s correlation coefficients (r), and then to Fisher’s z scale with a corresponding standard error (Borenstein et al., 2009; Gurevitch et al., 2018; Lipsey & Wilson, 2001). For unstandardized regression coefficients and odds ratios, we transformed the effects into standardized coefficients and then applied the same conversion procedure.⁶ The full coded dataset of 183 common effect estimates is more than sufficient to conduct a valid meta-analysis and related moderator tests (Jackson & Turner, 2017; Matthes et al., 2019; Rains et al., 2018).⁷

Second, to complement our hypothesis-testing results using this standard approach in the literature of analyzing common effect estimates coded from multivariate analyses, we also analyze a more limited subset of zero-order correlations between PE-OnPP and PE-OffPP (k = 45) that were also reported in the studies that met our selection criteria (often reported in correlation matrices in appendices). An advantage of analyses based on this smaller subset of zero-order correlation estimates is a more precise measurement of the PE-OnPP and PE-OffPP relationship, while a disadvantage is that this much smaller sample size increases the likelihood that the moderator analyses used to test H2 (Over-time convergence) and H3 (Democratic context) may yield non-significant findings due to insufficient sample size rather than the absence of an actual moderator effect. We therefore report hypothesis-testing results based on separate analyses of these two samples (i.e., k = 183 for the common effect estimate dataset, and k = 45 for the zero-order correlation dataset), and synthesize findings from both sets of analyses to interpret the implications of the findings.

To analyze the data, we conduct a multilevel random effects meta-analysis of precise effect size. Conventional random effects meta-analyses are based on a two-level model of data samples (level 1) from which researchers obtain coded effects (level 2), and effect sizes at level 2 are assumed to be independent (Assink & Wibbelink, 2016; Borenstein et al., 2009). Multilevel random effects meta-analyses add a third level in order to account for known dependencies among multiple effects, such as effects coded from the same study or dataset (see, for example, Dinesen et al., 2020; Matthes et al., 2019). The multilevel structure of our coded dataset is produced by the main criterion for inclusion, namely that the study used data from the same dataset to analyze at least two relationships: the association between PE and OnPP, and the association between PE and OffPP. A multilevel meta-analysis approach is therefore necessary to properly account for the nested structure of the data. Specifically, we use the “meta” package in R (Schwarzer, 2022; Schwarzer et al., 2015) to first fit a random effects model using the meta::metagen function, and specify the Knapp-Hartung Sidik Jonkman (HKSJ) method as estimator. We then fit a multilevel random effects model using the metafor::rma.mv() function (Viechtbauer, 2010, 2021), specifying that the analysis is clustered by dataset. In addition to our use of multilevel random effects meta-analysis to analyze whether the relationship between PE-OnPP is weaker than PE-OffPP (H1), we test the two additional hypotheses related to over-time convergence (H2) and democratic context (H3) via moderator analyses in separate multilevel models for online and offline political participation. We conclude the analysis by conducting a series of robustness tests and sensitivity analyses that confirm the reported findings.

Meta-analytic Sample and Descriptive Statistics

The 48 studies that met our selection criteria were published between 2002 and 2019, and analyzed 41 distinctive datasets based on surveys conducted between 2000 and 2016. Based on the selection criteria, each study contained at least two relevant effects: the relationship between political efficacy as an independent variable, and both OnPP and OffPP as separate dependent variables. Most of the studies analyzed cross-sectional data; only eight of the identified studies included repeated-wave panel data, and six employed a lagged model.

To assess whether systematic differences in measurement can be identified in the studies’ measures of OnPP and OffPP, we tested whether there was a statistically significant difference between the reported Cronbach’s alpha values for the two forms of participation. The t-test is not statistically significant for the common effect estimate dataset (k = 183) or for the zero-order correlation dataset (k = 45), thus indicating no systematic difference in measurement error between the OnPP and OffPP measures (see Supplementary Material for tabular output).

Table 1 summarizes additional characteristics of the studies that met our selection criteria. Consistent with findings in the digital participation literature (e.g., Boulianne, 2019), about half of the studies (n = 22) analyzed data from US samples, while the remaining studies used data from countries across the globe. Of the studies using data outside the United States, eight samples were from China, six from South Korea, two from Germany, and two from Singapore. Samples from nine other countries (Czech Republic, Denmark, Finland, Hong Kong, Israel, Kuwait, Italy, Taiwan, and the United Kingdom) were used only in one study. Two studies compared data from more than one country (Chan et al., 2017; Saldaña et al., 2015),⁸ and two studies analyzed pooled data from more than one country (Adugu & Broome, 2018; Huber et al., 2019).⁹

Table 1. Sample characteristics.

Characteristics of studies and effects	Number of Studies	Number of Effects
Type of sample
Online panel of individuals adjusted to match census data	27	89
University students	10	36
Random sample RDD or cellphone	7	46
Other type of convenience sample (e.g., twitter users survey)	4	13
Total	48	184
Country
Outside the United States	27	93
United States	22	91
Total	49	184
Sample size
Less than 249 respondents	1	2
250 to 499 respondents	18	71
500 to 749 respondents	8	26
750 to 999 respondents	5	15
1000 to 1249 respondents	8	27
1250 to 1499 respondents	4	13
More than 1500 respondents	9	30
Total	53	184

Note. For country, the total number of studies documented in Table 1 (n = 49) exceeds the total number of studies in the meta-analytic dataset (n = 48) because one study (Saldaña et al., 2015) analyzed two separate samples: one of the United States, and one outside of the US (the United Kingdom). For sample size, the total number of studies by sample size (n = 53) exceed the total 48 studies in the meta-analytic dataset because several studies include multiple models based on different sample sizes (AlSalem, 2015; Guidetti et al., 2016; Jugert et al., 2013; Saldaña et al., 2015).

Results

Testing H1: Efficacy Hypothesis

The results of the random effects meta-analysis are depicted in the forest plot in Figure 1. This figure shows the effect sizes for each study’s common effect estimates, listed in ascending order of effect size. The effect size for OnPP (Figure 1a) is 0.123, which is significantly different from zero (95% CI: 0.099, 0.147). The effect size for OffPP (Figure 1b) is 0.146, which is also significantly different from zero (95% CI: 0.117, 0.175). As noted at the bottom of Figure 1b, the overall estimated effect size for the model as a whole is 0.135 (95% CI: 0.116, 0.154). Taken together, results in the forest plots of the random effects meta-analysis show that political efficacy is positively associated with both online and offline participation, and the point estimate of the effect size for offline participation is larger than that for online participation. However, when confidence intervals are taken into account, the findings in Figure 1 suggest that there is no statistically significant difference in the magnitude of the association between PE with OnPP versus OffPP.

Figure 1a. OnPP forest plot, random effects meta-analysis (H1: efficacy hypothesis).

Figure 1b. OffPP forest plot, random effects meta-analysis (H1: efficacy hypothesis).

The results of the random effects meta-analysis presented in the forest plots in Figure 1 do not yet account for the nested structure of our coded meta-analytic dataset. As noted, to properly account for the dependent nature of the coded effects, we estimated a multilevel (i.e., three-level) random effects meta-analysis, which allows for the valid inclusion of multiple coded effects from the same dataset. The results of the multilevel model that tests for subgroup differences between OnPP and OffPP reported in Table 2 are consistent with results in the forest plot in Figure 1. Both online and offline participation have a positive association with political efficacy that is significantly different from zero. The test of subgroup differences shows that there is no statistically significant difference in the associations between political efficacy and online versus offline participation for the common effect estimate dataset of k = 183 (Table 2) or for the zero-order correlation dataset of k = 45 (Table 2). Finally, the results comparing the model fit of the three-level random effects model that clusters at the third level by dataset compared to the standard two-level random effects model confirm improved fit for the three-level model (p < .001).

Table 2. Test for subgroup differences in PE-OnPP versus PE-OffPP (H1, Efficacy hypothesis).

a. Common effect estimate dataset (k = 183)
Category	S	k	b	CI	F	p-value	% var. level 1	% var. level 2	% var. level 3
Offline PP (intercept)	41	183	0.154	[0.121, 0.187]	3.400 (1,181)	0.000	6.029	53.612	40.359
Online PP			−0.019	[−0.039, 0.001]		0.067
b. Zero-order correlation dataset (k = 45)
Category	S	k	b	CI	F	p-value	% var. level 1	% var. level 2	% var. level 3
Offline PP (intercept)	14	45	0.38	[0.301, 0.460]	0.769 (1,43)	0.000	5.766	62.874	31.360
Online PP			−0.027	[−0.089, 0.035]		0.385

Note. Models estimated as multilevel random effects meta-analysis; % var. at level 1 is due to sampling error; at level 2 is due to within-cluster variance (i.e., variance within the same dataset); and at level 3 is between-cluster variance (i.e., between distinctive datasets in the meta-analytic sample). While the variance distribution results from within-cluster variance (k = 183, 53.6%; k = 45, 62.9%) and between-cluster variance (k = 183, 40.4%; k = 45, 31.4%), the test of subgroup differences for both datasets shows no statistically significant difference in the associations between political efficacy and online versus offline participation (for k = 183, p = 0.067; for k = 45, p = 0.385).

Taken together, the results do not support the efficacy hypothesis (H1). Based on the best available evidence analyzed by the most robust analytical methods, there is no statistically significant difference in the size of the positive association between efficacy and online participation and the size of the positive association between efficacy and offline participation.

Testing H2: Over-time Convergence

The findings for H1 indicate there is no statistically significant difference in the modestly positive effects between PE and OnPP versus PE and OffPP when the analysis does not account for the year in which each study’s survey was conducted. Thus, the over-time convergence hypothesis would be supported if a statistically significant difference was evident in the effect size of our key variables (between PE and OnPP compared to PE and OffPP) only in the early years of the observation period. As noted, the expectations discussed in the literature (e.g., Farrell, 2012; Ohme et al., 2018) suggest that if a statistically significant distinction was apparent in the early years of the sample, it would be due to a weaker relationship between PE and OnPP than between PE and OffPP – but that the association between PE and both types of participation would become more similar in recent years.

To test the hypothesis of over-time convergence (H2), we assessed whether there is an interaction effect in the multilevel random effects meta-analysis framework between year and the dependent variable type. The findings reported in Table 3 for both the common effect estimate dataset of k = 183 and the zero-order correlation dataset of k = 45 provide no evidence for moderation between year and the dependent variable type (i.e., OnPP and OffPP), as the interaction between dependent variable type and year is not significant. This lack of a systematic shift in the effect size for OnPP and OffPP over time is also supported by forest plots with effects ordered by year in which the relevant survey was conducted, which shows no clear over-time pattern (see Supplementary Material). Thus, we find no evidence of an over-time convergence in the magnitude of the associations between political efficacy and the two types of political participation (OnPP and OffPP). The results therefore reaffirm the main conclusion about efficacy in relation to these two types of participation (H1): there is no significant difference in the magnitude of the positive associations between political efficacy and OnPP versus OffPP, and these relationships are stable over time.

Table 3. Test of the effect of year as moderator (H2, Over-time convergence).

a. Common effect estimate dataset (k = 183)
Category	S	k	b	CI	F	p-value
Intercept (OffPP)	41	183	−11.328	[−30.415, 7.758]	1.849 (1,179)	0.243
Online PP			8.584	[−5.613, 22.780]		0.234
Year			0.006	[−0.004, 0.015]		0.237
Online PP * year			−0.004	[−0.011, 0.003]		0.233
b. Zero-order correlation dataset (k = 45)
Category	S	k	b	CI	F	p-value
Intercept (OffPP)	14	45	39.413	[−13.809, 92.636]	1.097 (1,41)	0.142
Online PP			−34.324	[−95.130, 26.482]		0.261
Year			−0.019	[−0.046, 0.007]		0.146
Online PP * year			0.017	[−0.013, 0.047]		0.261

Note. Models estimated as multilevel random effects meta-analysis

Testing H3: Democratic Context

To test the democratic context hypothesis, we used data from the Varieties of Democracy (V-Dem) index of electoral democracy (v2x_polyarchy; Coppedge et al., 2020a, pp. 42–43, 2020b; Teorell et al., 2019), which measures the extent to which the ideal of electoral democracy is achieved in its fullest sense. This macro-level index assesses the degree to which governing systems are responsive to their citizens, based on aggregating lower-level indices that assess factors such as electoral competition and freedom of expression (Coppedge et al., 2020a, p. 288). The index score assigned to each coded estimate in the dataset corresponds to the country’s score on the electoral democracy index for the year in which the survey was conducted.¹⁰

The findings presented in Table 4 show that this polyarchy summary measure of the level of electoral democracy is not a statistically significant moderator of either the relationship between efficacy and OnPP or the relationship between efficacy and OffPP, for either the common effect estimate dataset or for the zero-order correlation dataset. However, we also analyzed the more specific lower-level indices that inform the V-Dem overall index of electoral democracy, including measures of freedom of association, clean elections, freedom of expression, and elected officials. In addition, we tested additional contextual measures that may also act as moderators, namely civil society strength (Coppedge et al., 2020a, p. 288), Internet penetration (Robinson et al., 2015), and globalization as measured by trade (World Bank, 2021).

Table 4. Test of democratic contextual features as moderators (H3, Democratic context).

a. Common effect estimate dataset (k = 183)
Moderator	Participation	S	k	b	CI	F	p-value
Polyarchy	Online PP	39	78	−0.009	[−0.107, 0.089]	0.035 (1,83)	0.853
	Offline PP	39	87	−0.111	[−0.229, 0.007]	3.500 (1,92)	0.065
Freedom of association	Online PP	39	78	0.001	[−0.088, 0.090]	0.001 (1,83)	0.974
	Offline PP	39	87	−0.098	[−0.205, 0.010]	3.274 (1,92)	0.074
Clean elections	Online PP	39	78	−0.003	[−0.089, 0.083]	0.005 (1,83)	0.946
	Offline PP	39	87	−0.111	[−0.214, −0.007]	4.515 (1,92)	0.036
Freedom of expression	Online PP	39	78	−0.023	[−0.120, 0.074]	0.230 (1,83)	0.633
	Offline PP	39	87	−0.130	[−0.243, −0.017]	5.181 (1,92)	0.025
Elected officials	Online PP	39	78	−0.007	[−0.083, 0.070]	0.031 (1,83)	0.860
	Offline PP	39	87	−0.092	[−0.184, 0.000]	3.936 (1,92)	0.050
Civil society strength	Online PP	39	78	−0.032	[−0.152, 0.089]	0.271 (1,83)	0.604
	Offline PP	39	87	−0.156	[−0.296, −0.015]	4.822 (1,92)	0.031
Internet penetration	Online PP	38	77	0.000	[−0.002, 0.002]	0.072 (1,82)	0.789
	Offline PP	38	86	−0.002	[−0.004, 0.001]	1.525 (1,91)	0.220
Globalization (trade)	Online PP	38	77	0.000	[0.000, 0.001]	1.622 (1,82)	0.206
	Offline PP	38	86	0.000	[0.000, 0.001]	0.035 (1,91)	0.853
b. Zero-order correlation dataset (k = 45)
Moderator	Participation	S	k	b	CI	F	p-value
Polyarchy	Online PP	14	20	0.039	[−0.170, 0.247]	0.151 (1,19)	0.702
	Offline PP	14	23	−0.035	[−0.311, 0.241]	0.070 (1,22)	0.794
Freedom of association	Online PP	14	20	0.026	[−0.164, 0.216]	0.083 (1,19)	0.777
	Offline PP	14	23	−0.057	[−0.311, 0.197]	0.215 (1,22)	0.647
Clean elections	Online PP	14	20	0.029	[−0.144, 0.202]	0.123 (1,19)	0.730
	Offline PP	14	23	−0.043	[−0.274, 0.188]	0.147 (1,22)	0.705
Freedom of expression	Online PP	14	20	0.033	[−0.165, 0.231]	0.121 (1,19)	0.732
	Offline PP	14	23	−0.030	[−0.294, 0.235]	0.054 (1,22)	0.819
Elected officials	Online PP	14	20	0.023	[−0.138, 0.183]	0.087 (1,19)	0.771
	Offline PP	14	23	−0.053	[−0.268, 0.162]	0.259 (1,22)	0.616
Civil society strength	Online PP	14	20	0.041	[−0.206, 0.289]	0.123 (1,19)	0.730
	Offline PP	14	23	−0.023	[−0.354, 0.307]	0.022 (1,22)	0.885
Internet penetration	Online PP	14	20	0.001	[−0.004, 0.006]	0.297 (1,19)	0.592
	Offline PP	14	23	0.000	[−0.007, 0.007]	0.004 (1,22)	0.948
Globalization (trade)	Online PP	14	20	0.000	[−0.001, 0.001]	0.005 (1,19)	0.944
	Offline PP	14	23	0.000	[−0.002, 0.001]	0.251 (1,22)	0.621

Note. Models estimated as multilevel random effects meta-analysis

The findings on the relationship between political efficacy and OnPP reveal that none of these more specific country-level measures are significant moderators for the common effect estimate dataset (Table 4) or for the zero-order correlation dataset (Table 4). The same null effects findings pattern holds for PE-OffPP for the zero-order correlation sample, which is the smaller of the two data samples (k = 45). However, for the larger common effect estimate dataset (k = 183), the findings do show that three of these country-level features moderate the relationship between political efficacy and OffPP: countries with cleaner elections, greater freedom of expression, and stronger civil society have, on average, a weaker relationship between political efficacy and OffPP. This means that, according to results based on the common effect estimate dataset, some of the association between political efficacy and political participation is explained by the strength of these contextual democratic features for OffPP but not OnPP. For offline participation, these findings suggest that in contexts that have stronger democratic institutions, individual-level political efficacy may play less of an important role to motivate individuals to engage in offline opportunities of participation, though further research would be needed to establish the causal mechanisms at play.

For online participation, however, the implications of the tests for country-level moderators for both the common effect estimate dataset (k = 183) and the zero-order correlation dataset (k = 45) are clear: the positive association between political efficacy and OnPP is of similar magnitude as the main effect for OffPP, and is unaffected by all tested measures of country context. This indicates that online participation’s positive association with people’s sense of their ability to engage in politics and impact political processes is stable across the globe in diverse political contexts.

Robustness Tests

This section summarizes a series of robustness tests, including sensitivity analyses, study-level robustness tests, and publication bias tests. All results summarized in this section are documented in further detail in the Supplementary Material and replication files.

Sensitivity Analyses

We conducted several sensitivity tests to assess whether the findings are robust despite the varying specifications of the original studies. These sensitivity analyses all support our main findings, including testing for potential effects of study quality, sample features, and outlier effects. For study quality, we tested whether objective measures of study quality affect the results by testing for differences between findings from studies published in high-ranked versus low-ranked journals and unranked studies; the results showed no statistically significant difference.¹¹ Regarding features of the study sample, sample size is a significant moderator for some models (i.e., OnPP k = 45 and OffPP k = 183), but the coefficient is modest in size, and the main findings hold. Sample representativeness is a significant moderator in only one model (OffPP k = 183), with a modest coefficient, and the main findings hold. As a final sensitivity test, standard tests for outlier detection identified no effects that met established criteria to qualify as outliers for either the common effect estimate dataset or the zero-order correlation dataset (Cook & Weisberg, 1982; Viechtbauer & Cheung, 2010).

Study-Level Robustness Tests

For the common effect estimate dataset (k = 183), we examined whether the main findings are affected by attitudinal control variables in the multivariate models from which the effects were coded, namely the key attitudes in the literature of political knowledge and political interest. Political interest was not a significant moderator for either type of participation. Political knowledge was not a significant moderator for OffPP, but the findings showed a marginally significant association with OnPP (b = −0.044, p = .049), indicating a weaker relationship between political efficacy and OnPP when political knowledge is taken into account. Yet, even with the inclusion of political knowledge in the models, the main findings hold of no significant difference in the magnitude of the positive association between political efficacy and the two types of political participation. We conducted several additional study-level robustness tests for both the common effect estimate dataset and the zero-order correlation dataset, including assessment of distinctive types of efficacy, findings for the United States compared to all other countries, and whether OnPP is defined as social media activities. The robustness tests show that the main findings hold when accounting for these study-level factors, and the findings are reported in full in the Supplementary Material.

Publication Bias

We conducted standard publication bias tests (Egger et al., 1997; Sterne & Egger, 2005). The results provide evidence of modest publication bias for OffPP but no evidence of such bias for OnPP, although there is not yet consensus regarding thresholds for publication bias at the effect-level in multilevel meta-analysis (Fernández-Castilla et al., 2021; Rodgers & Pustejovsky, 2021). In addition, we also tested for publication bias by conducting a p-curve analysis (Carter et al., 2019; Simonsohn et al., 2014; Sun & Pan, 2020) and the results show no indication of publication bias.

Robustness Tests Summary

In sum, the results of these robustness tests confirmed the main findings: results based on both the common effect estimate dataset (k = 183) and the zero-order correlation dataset (k = 45) show that political efficacy is positively related to both online and offline political participation; these two positive relationships are similar in magnitude; there is no statistically significant difference in the strength of the associations between PE and OnPP versus OffPP across time periods; and the positive association between efficacy and OnPP is unaffected by country contextual features. Finally, according to the larger common effect estimate dataset (k = 183), the findings suggest that several country context features may moderate the relationship between efficacy and OffPP, but the smaller zero-order correlation dataset (k = 45) does not yield significant findings for any of the democratic context moderators.

Discussion

This study provides the most comprehensive investigation to date of the important question of how online and offline political participation relates to people’s sense of their capacity to engage in and impact political processes. Both normative and empirical studies have raised the question of whether online political participation is considered a meaningful form of political activity (Christensen, 2011; Freelon et al., 2020; Halupka, 2014; Karpf, 2012; Matthews, 2021). This type of activity has been considered in some research and public discourse as a low-cost way to participate that may also be used to express frustration with no expectation of political change, and thus may even demobilize citizens who would otherwise have been politically engaged (Anduiza et al., 2012; Gladwell, 2010). Despite these discourses, the current study shows that believing in one’s ability to participate in and impact political processes is as strongly related to online as to offline forms of political participation. The meta-analysis findings do not support the main efficacy hypothesis (H1), as political efficacy has a modestly positive association with both online and offline participation, with no statistically significant difference in the magnitude of these associations.

The empirical challenges involved in addressing this question are considerable due to limited availability of high-quality cross-national and longitudinal data on online participation, which limits researchers’ capacity to make valid inferences on the relation between key variables over time. The current study demonstrates how a multilevel random effects meta-analysis of all relevant empirical studies can yield new evidence to inform this important debate. Focusing on the online-offline distinction entails the theoretically and empirically challenging task of definitively distinguishing between these two categories of participation. Thus, the research design used the common distinction in the literature, which allowed the most robust test possible of the difference between the strength of the PE-OnPP association and the PE-OffPP association, and of whether this difference varied over time and across diverse country contexts. The results showed that a positive and substantively similar relationship between political efficacy and OnPP versus OffPP remained stable over time and that the strength of the association between PE and OnPP is stable across diverse country contexts. As noted, the moderator results for several features of democratic context (i.e., clean elections, freedom of expression, and civil society strength) were statistically significant in analyses of the common effects estimate dataset (k = 183), but not in analyses of the zero-order correlation dataset (k = 45), suggesting the importance of future research on these topics as more zero-order correlation data become available. To this end, we support Amsalem & Nir’s, 2021, p. 636) strong recommendation that documentation of correlation matrices of all study variables become standard practice for appendices of multivariate quantitative studies. Particularly for topics such as online political participation for which large-scale comparative surveys will continue to include a limited number of indicators for the foreseeable future, the accumulation of scientific knowledge will be greatly facilitated through meta-analyses of the growing body of country-specific research.

As noted in the descriptive statistics for the sample, the current analysis includes few studies that analyzed repeated-wave panel data. Thus, an important limitation of the current study is the lack of data that can be used to assess the causal direction of the relationship between political efficacy and political participation. Although conventional wisdom in this field tends to presume that political efficacy has a causal effect on political participation, research on offline participation suggests it is important to consider the possibility that the causal arrow can also be reversed, such that political participation enhances efficacy (Finkel, 1985; Quintelier & van Deth, 2014).

Given this possibility, we conclude by situating the current study within a broader stream of research that investigates the core topics of causal inference from several angles. The meta-analytic sample analyzed in the current study includes eight studies that analyzed repeated-wave data (six of which conducted lagged analyses), and additional repeated-wave studies on these topics have been emerging rapidly, which will enable future meta-analytic study of causal direction. While the comprehensive meta-analytic findings of the current study definitively establish that political efficacy has a similar positive association with both online and offline political participation, more fine-grained research is needed to assess the underlying mechanisms that may enhance these relations through different activities, and in different contexts. For example, future research is needed to assess whether the specific types of political acts within the broad OnPP/OffPP distinction of the current meta-analysis (e.g., engagement with the news, connecting with politicians online, or joining online groups) are more highly associated with individuals’ sense of efficacy.

To this end, multi-method research is needed on the relationship between political efficacy and political participation in the digital era, including studies based on cross-national repeated-wave surveys, qualitative fieldwork, and experimental data. For example, Shuman et al.’s (2021) experimental study which shows that collective action that is non-normative and nonviolent succeeds in motivating support for policy goals can be adapted to test the validity of these findings for online forms of participation, and in contexts characterized by varying levels of political efficacy. Further, a high-quality and large-scale cross-national survey specifically designed to test the effects of macro-level context will be useful for precisely assessing contextual effects through multilevel modeling and cross-level interaction tests. Building on the current study’s clear affirmation of the positive association between political efficacy and online participation, these future lines of research on mechanisms that underlie this association can shed new light on the important question of how different types of online political participation may affect the connection between citizens and democratic processes in the digital era.

Open Scholarship

This article has earned the Center for Open Science badges for Open Data, Open Materials and Preregistered. The data and materials are openly accessible at http://doi.org/10.17605/OSF.IO/AF5DR. The pregistration is available at http://doi.org/10.17605/OSF.IO/MHXA8.

Acknowledgments

We are grateful for expert input on the development of this study to (in alphabetical order) Jeremy Albright, Johanna Dunaway, Caleb Scheidel, Wolfgang Viechtbauer, and Alon Zoizner. We are solely responsible for the research.

Disclosure Statement

No potential conflict of interest was reported by the author(s).

Supplementary Material

Supplemental data for this article can be accessed on the publisher’s website at https://doi.org/10.1080/10584609.2022.2086329

Data Availability Statement

The data described in this article are openly available in the Open Science Framework at http://doi.org/10.17605/OSF.IO/AF5DR.

Additional information

Funding

This research was funded by the Israel Science Foundation [grant 1246/20 to Jennifer Oser]; and by the European Research Council [grant 335607 to Eran Halperin].

Notes on contributors

Jennifer Oser

Jennifer Oser is an Associate Professor in the department of Politics and Government at Ben-Gurion University in Israel. Her research focuses on the relationships between public opinion, political participation, and policy outcomes.

Amit Grinson

Amit Grinson completed his M.A. in Sociology and Anthropology at Ben-Gurion University in Israel. He has researched topics of trust and online participation, and his methodological research interests include data visualization and data analytics.

Shelley Boulianne

Shelley Boulianne is an Associate Professor in Sociology at MacEwan University in Canada. She researches citizens’ engagement in civic and political life, including topics related to digital media, climate change, environmental policies, and misinformation.

Eran Halperin

Eran Halperin is a Full Professor of Psychology at the Hebrew University of Jerusalem in Israel. His research uses psychological and political theories and methods to investigate different aspects of inter-group conflicts and emotion regulation.

Notes

1. See registered hypotheses and research design in the Open Science Framework (OSF) http://doi.org/10.17605/OSF.IO/MHXA8.

2. All analyses were conducted in R version 4.1.1. Open Science Framework: http://doi.org/10.17605/OSF.IO/AF5DR.

3. The search string consisted of the following Boolean string: ALL (“political efficacy” OR “general efficacy” OR “internal efficacy” OR “external efficacy” AND part* AND online OR e-participation OR “digital media” AND offline).

4. The seminars and conference talks were all conducted virtually: University of Montreal Seminar, Canada Research Chair in Electoral Democracy and the Research Chair in Electoral Studies on April 21, 2020; European University Institute’s Political Behavior Colloquium, May 19, 2020; American Political Science Association Political Communication Pre-Conference, September 1, 2020; International Journal of Press/Politics, September 8, 2020.

5. One of the effects reported in the descriptive statistics (from Kim & Baek, 2018) was an outlier that was too large to transform to the common effect estimate measure of Fisher’s z. The sample size for the common effect estimate dataset is therefore k = 183 for all analyses.

6. To transform unstandardized regression coefficients, we multiplied the unstandardized coefficients by the standard deviation of the predictor variable, divided by the standard deviation of the outcome variable. For odds ratios, we used Menard’s second formula (Menard, 2004, p. 219).

7. As clarified by Jackson and Turner (2017), five or more studies are all that is needed to achieve sufficient power in a random effects meta-analysis. In a meta-analysis of meta-analyses conducted over 60 years in the field of communication, Rains et al. (2018) find that the average effect estimate of published studies is 49.

8. Specifically, Chan et al. (2017) compared data from three countries (China, Hong Kong, and Taiwan); and Saldaña et al. (2015) compared data from two countries (the United Kingdom and the United States).

9. Specifically, Adugu and Broome (2018) analyzed data from five countries (Jamaica, Trinidad and Tobago, Guyana, Surinam, and Haiti); Huber et al. (2019) analyzed data from 19 countries (Argentina, Brazil, China, Estonia, Germany, Indonesia, Italy, Japan, South Korea, New Zealand, the Philippines, Poland, Russia, Spain, Taiwan, Turkey, Ukraine, United Kingdom, and the United States).

10. As noted in the description of sample characteristics, two studies aggregate data from several countries (Adugu & Broome, 2018; Huber et al., 2019) and therefore cannot be included in analyses that test for country-level moderators.

11. Journal ranking was coded according to the four quartiles in Web of Science’s Journal Citation Reports (JCR), and a “not ranked” category, which includes both peer reviewed sources not ranked in JCR, and unpublished sources (e.g., conference papers, dissertations). JCR rankings were accessed through Clarivate Analytics (jcr.clarivate.com) and were coded according to the year in which the study was published.