Does Internet Use Affect Engagement? A Meta-Analysis of Research

Abstract

Scholars disagree about the impact of the Internet on civic and political engagement. Some scholars argue that Internet use will contribute to civic decline, whereas other scholars view the Internet as having a role to play in reinvigorating civic life. This article assesses the hypothesis that Internet use will contribute to declines in civic life. It also assesses whether Internet use has any significant effect on engagement. A meta-analysis approach to current research in this area is used. In total, 38 studies with 166 effects are examined. The meta-data provide strong evidence against the Internet having a negative effect on engagement. However, the meta-data do not establish that Internet use will have a substantial impact on engagement. The effects of Internet use on engagement seem to increase nonmonotonically across time, and the effects are larger when online news is used to measure Internet use, compared to other measures.

Keywords:

• Internet
• political participation
• meta-analysis

Internet use has come under attack by several scholars who argue that people are surfing the Internet instead of engaging in civic and political activities. Several studies have examined the impact of Internet use on civic and political engagement. These studies have produced conflicting findings about whether the Internet will have a positive or negative effect as well as whether Internet use has a substantial effect on engagement. The conflicting findings have fueled debates about whether the Internet will contribute to declines in civic life. This article evaluates the findings of 38 studies (with 166 effects) to assess whether Internet use has a negative effect or a positive effect on civic and political engagement. The article combines quantitative and qualitative analyses of the existing research, offering potential explanations of the conflicting findings.

Theories of Internet Use and Engagement

Several theories compete to predict how the Internet will affect civic and political engagement. One set of scholars, including Putnam (1995, 2000), believe the Internet will have a detrimental impact on engagement, because this technology is being used primarily for entertainment. As a result of these distractions, citizens may have less time to devote to civic or social activities, such as joining civic groups, visiting family and friends, or bowling in leagues (Putnam, 1995, 2000). In response to this theory, scholars point out that the most common uses of the Internet are for social interaction and information-searching. The most common Internet use is for e-mail (Day, Janus, & Davis, 2005; National Telecommunications and Information Administration [NTIA], 2004; Nie & Erbring, 2000). The second most common use is searching for information (NTIA, 2002, 2004). These uses are far more likely than other uses; a 15% to 20% frequency gap separates these uses from other Internet activities (NTIA, 2002, 2004). Trend data suggest that citizens are increasingly using the Internet to follow news and political campaigns and to engage in online political activities such as donating (Howard, 2006). However, this response has not appeased the critics. As such, the present meta-analysis will focus on the hypothesis that Internet use has a negative impact on civic and political engagement.

Another set of scholars argue that the Internet will have positive impacts on civic and political engagement. Within this set of scholars, there are at least two groups. One group argues that the Internet will serve to activate those citizens who are already predisposed or interested in politics (Bimber, 1999; Bonfadelli, 2002; DiMaggio, Hargittai, Celeste, & Shafer, 2004; Hendriks Vettehen, Hagemann, & Van Snippenburg, 2004; Krueger, 2002; Norris, 2001; Polat, 2005; Weber, Loumakis, & Bergman, 2003). The Internet reduces the costs (time, effort) of accessing political information and offers more convenient ways of engaging in political life (e.g., online petitions). These opportunities are attractive to those people who are interested, knowledgeable, and already engaged in the political process (see Figure 1, path c). Because the predictors of Internet use are similar to the predictors of engagement, the benefits of the Internet will be restricted to those who are engaged (see Figure 1, paths a and b). This argument tends to rely on “common cause” thinking in assessing the implications of the Internet, rather than testing the relationship between Internet use and engagement. Norris (2000) exemplifies this position (see chapter 6). Her theory of virtuous circle, which speaks to media use in general, posits that media use will serve to activate the engaged rather than mobilizing new participants to become involved in the political process.

Figure 1. Theoretical positive effects of Internet use and political engagement.

Another group argues that the Internet could mobilize politically inactive populations (Barber, 2001; Delli Carpini, 2000; Krueger, 2002; Ward, Gibson, & Lusoli, 2003; Weber et al., 2003). The convenience of the Internet may entice a broader set of citizens to engage in politics. Increased information access may reduce knowledge deficiencies that are used to excuse disengagement. The improved access to information may reduce knowledge differences observed between those of high socioeconomic status versus those of low socioeconomic status, men versus women, and youth versus other age groups (Delli Carpini & Keeter, 1996). New online opportunities for expression may help with the identification and organization of like-minded citizens, expanding engagement across diverse populations. The convenience or novelty of online engagement may draw in those disillusioned with traditional methods of political participation. The strongest proponents of this theory focus on the potential of the Internet to affect young people's levels of engagement (e.g., Delli Carpini, 2000). This group is the most likely of all age groups to be online. They are highly skilled and intense users of this medium, increasing the potential for a significant effect of Internet use on engagement.

Both groups agree that the Internet may reinvigorate civic life by increasing access to political information, facilitating political discussion, developing social networks, and offering an alternative venue for political expression and engagement (Polat, 2005; Ward et al., 2003). Both groups challenge the view that the Internet will contribute to civic decline. Using a meta-analysis approach, this article will evaluate whether this optimism is justified given the findings from 38 studies. The meta-analysis assesses whether Internet use decreases engagement and attempts to assess whether the effect is significant.

Scope and Methodology

Selection of Studies

There is relatively little research on the relationship between Internet use and political engagement. As such, in assessing the existing literature, an attempt was made to identify all data sources and studies, rather than select a sample of studies for inclusion. One of the major criticisms of meta-analysis is the focus on published studies, which can result in a bias toward finding significant effects (Lipsey & Wilson, 2001). In other words, studies that find no effects tend to go unpublished and thus would not be included in the meta-analysis. As a result of this publication bias, the estimated effects may be upwardly biased. To address this bias, I used the published studies to identify additional relevant studies (e.g., conference papers). In addition, I contacted researchers of published pieces to identify unpublished pieces, which also helped to address this bias. The study identification approach helps identify a more robust set of studies for analysis.

The meta-analysis focuses on studies of the United States. I chose this geographic focus because of the volume of research on the American population. Almost all of the research in this meta-analysis is based on American respondents, but two studies combine Canadian and American respondents, so Canadians are included in some of the analysis (Quan-Haase, Wellman, Witte, & Hampton, 2002; Wellman, Quan-Haase, Witte, & Hampton, 2001). The reason for this geographic focus is to control for exogenous variables, which may affect the observed relationship between Internet use and political engagement. For example, international studies would introduce differences in political culture, political institutions, and political processes related to key political behavior (e.g., voter registration process, predetermined election dates). These exogenous variables could affect the observed relationship between Internet use and engagement.

I included both political and civic forms of engagement. ¹ Political engagement includes behaviors that directly relate to political institutions and the work of political institutions. As such, this conception encompasses voting, donating money to a campaign or political group, working for a political campaign or political group, and attending meetings or a rally for a candidate. This conception also includes various forms of political expression (e.g., wearing a button, talking politics, and writing a letter to the editor). This definition includes perceived behavioral changes as a result of Internet use (Johnson & Kaye, 2003) and perceptions about time spent in the community (Kraut et al., 2002). This definition does not include political knowledge, political interest, or attitudinal variables. Unfortunately, some studies did not allow for clear distinctions between behaviors and attitudes. For example, Moy, Manosevitch, Stamm, and Dunsmore (2005) as well as Kraut et al. (2002) examine feelings of community belonging as part of their index that includes behaviors.

In addition, I included nontraditional forms of political participation, such as signing petitions and participating in protests, marches, or rallies, but not other forms, such as political consumerism. ² The literature's tendency to use indexes results in blurring the lines between civic and political engagement. As such, I included activities that may be viewed as more civic forms of participation that may not be directly tied to political institutions. These activities include volunteering, working with others to solve a community problem, and serving in a local organization. I think that it is plausible that the relationship between Internet use and engagement may differ for civic and political forms of engagement (or different political activities). However, this meta-analysis is not capable of untangling these differing effects, because of researchers' tendencies to aggregate different activities into a single index.

To offer clarity on the independent and dependent variables, I included articles that assess engagement as “offline” activities. As a result, several studies are omitted such as Tian (2006) and Kobayashi, Ikeda, and Miyata (2006), who assess online forms of engagement only. In addition, Krueger's (2002, 2006) and Best and Krueger's (2005) findings about online mobilization and online participation are not summarized in this meta-analysis. Despite the intent to provide clarity on the independent and dependent variables, some studies did not permit this clarity, because online forms of engagement were aggregated into an index with offline forms of engagement. For example, Kim, Jung, Cohen, and Ball-Rokeach (2004); Quan-Haase et al. (2002); Wellman et al. (2001); and Weber et al. (2003) include measures of online political discussion in their index of political engagement. These authors do not isolate the effects of Internet use on offline engagement. Their measurement approach may bias the observed effects toward finding significant effects. This hypothesis cannot be directly tested, because the indexes also differ on many other dimensions.

The articles included in the meta-analysis included many different measurements of Internet use. For example, some studies compare Internet users and nonusers. Other studies examined hours of Internet use, years of experience using the Internet, or types of Internet use. Many studies used Web surveys as the mode of data collection and as a result could not compare Internet users and nonusers. These studies used the other measures of Internet use. Furthermore, many random-digit dialing (RDD) studies selected out Internet users (or those with Internet access) and used this subgroup for their analysis (Bimber, 2001; Krueger, 2002; Norris & Jones, 1998; Shah, Cho, Eveland, & Kwak, 2005) or a component of their analysis (Jennings & Zeitner, 2003; Kenski & Stroud, 2006). As a result of this selection approach, the findings between Web surveys and RDD studies can be compared and aggregated, because they share an analytic focus on Internet users. However, the implications of having a self-selected sample versus a more random sample become the subject of a forthcoming discussion about differences in the observed effects.

To move beyond the “common cause” approach of assessing the implications of the Internet (Figure 1, paths a and b), this meta-analysis focuses on multivariate analyses. All of the studies control for variables, such as education, that predict both Internet use and engagement. The net effect (controlling for common causes) is the focal point of the meta-analysis.

Examining Effects

The analysis of meta-data focuses on three measures: the number of positive and negative effects, the number of statistically significant and insignificant effects, and the average effect size. ³ While the first two measures are fairly straightforward, the latter finding poses several challenges. The variation in reporting strategies makes it difficult to identify a common effect statistic—a common issue in meta-analysis research (Lipsey & Wilson, 2001). Ideally, means, bivariate covariances, or proportions are available alongside measures of variance (e.g., standard deviation; Lipsey & Wilson, 2001). The measures of variance are used in calculating weights, which are employed in calculating average effect sizes. However, these more complex estimation procedures were not possible in this meta-analysis, because almost all of the studies reported standardized coefficients and did not report variance (e.g., standard errors of the estimates). Using sample size as an alternative weighting variable is not appropriate for this set of studies, because the largest samples tend to be self-selected Web surveys. Because of the sampling issues with the large Web surveys, I opted to not weight the effects in a way that would give these studies a greater impact on the computed average effect.

As such, the calculation of the average effect is simply an average of the standardized coefficients for all studies where ordinary least squares regression was used (or assumed to be used, given the measurement of the dependent variables). The studies that used logistic regression analysis were not included in the computed average effect. There were too few of these studies (10 studies) to merit a separate calculation of an average effect. The estimate of the average effect is done cautiously, because of the limits of conducting meta-analyses on standardized regression coefficients and with studies with varying research designs (Lipsey & Wilson, 2001).

Meta-analysis of data involves calculating the average effects as though each observed effect is independent. However, in these studies, the effects within a study are clearly not independent. The issue of independence is further complicated in that multiple studies rely on the same data source. To address the multiple effects within a study, Lipsey and Wilson (2001) recommend averaging the effects within a study and using this average in calculating the average effect across studies. In addition, I calculated the average effect at the data source level. All three approaches are examined. However, the analytic focus is on the average of the individual coefficients, rather than effects averaged at the study or data source levels, because this approach enables an analysis of how the direction and significance of effects change depending on how Internet use is measured.

The discussion combines this quantitative review with a qualitative review of the literature. The qualitative analysis highlights methodological issues with the studies that may explain their observed pattern of effects. This qualitative review highlights issues related to sampling, methodology, measurement, and modeling of the causal process. In sum, this body of research does not lend itself to the ideal application of meta-analysis techniques; however, a qualitative analysis and cautiously pursued quantitative analysis allow for basic hypothesis testing about the presence of negative effects and whether the overall effect is substantial.

Findings

Table 1 provides a summary of the findings from 38 studies. Approximately 166 effects are tested. Approximately half of the effects are statistically significant. As an aggregate, the findings suggest that Internet use and political engagement are positively related; only 16% of the effects are negative (Table 1). When negative effects are observed, the effect size tends to be small. Approximately half of all negative effects are estimated as .02 or less, and most of the negative effects are below .05 (Figure 2). Only six negative effects from five different studies are statistically significant. Three of these six effects are contradictory to other research. As such, these significant, negative effects require some scrutiny.

Table 1 Aggregate findings on Internet use and engagement

	Significance	No.of effects	% of total effects
Positive effects	Statistically significant∗	74	45
	Not statistically significant	53	32
Negative effects	Statistically significant∗	6	4
	Not statistically significant	20	12
Direction not reported	Statistically significant∗	0	0
	Not statistically significant	13	8
Total		166	100
∗p < .05.

Figure 2. Distribution of effects of Internet use and political engagement.

Only one of the five studies is based on an RDD sample—a survey of Ann Arbor, Michigan, residents (Kwak, Skoric, Williams, & Poor, 2004). This study produced the lowest negative coefficient observed in Figure 2. This negative coefficient should be placed in perspective. The study reports many coefficients, and only one coefficient is negative and statistically significant. The significant, negative effect is derived from a comparison of nonusers, narrowband users, and broadband users, assuming these groups can be placed on a continuum of Internet adoption (Kwak et al., 2004). Kwak et al. (2004) find that broadband users are less engaged than narrowband users and nonusers (p < .05). This finding contradicts other studies that suggest that broadband users are slightly more likely to be engaged than nonusers (Krueger, 2002, 2006). ⁴ Many studies examine whether Internet users differ from nonusers (Best & Krueger, 2005; Jennings & Zeitner, 2003; Katz & Rice, 2002; Kenski & Stroud, 2006; Mossberger, Tolbert, & McNeal, 2008; Scheufele & Nisbet, 2002; Tolbert & McNeal, 2003). The findings tend to go unreported, and those findings that are reported do not involve a consistent definition of user (e.g., use anywhere versus use at home). These findings tend to suggest that Internet users, compared to nonusers, are more engaged, but the difference is not statistically significant (p > .05). ⁵ In sum, Kwak et al.'s (2004) findings related to negative effects contradict other studies examining users versus nonusers and broadband users versus nonusers.

Two other significant, negative effects were found. The directions of these effects are consistent with other research, but the sizes (and statistical significance) of the effects are not consistent with other research. These studies measure Internet use as household consumption, such as online banking (Moy et al., 2005) and recreational uses (Quan-Haase et al., 2002). Other studies also examine these types of uses and find small, negative effects that are not significant (Shah, Kwak, & Holbert, 2001; Shah, McLeod, & Yoon, 2001). The strong negative effect of household consumption on engagement is, along with Kwak et al.'s (2004) large negative effect, the coefficient at the extreme negative end of the distribution of effects (Figure 2). This study is an intercept survey of about 300 riders of the Seattle-area ferry system (Moy et al., 2005). This study also accounts for the highest positive effect in the distribution. ⁶

Finally, the last two significant, negative effects are from the same study. The standardized negative effects are .02. The two effects are based on a very large (self-selected Web) survey, which may shrink the standard error and produce statistically significant effects that are substantively insignificant. These effects are based on the analysis of the National Geographic Society (NGS) Web survey data. Gibson, Howard, and Ward (2000) found that the effect of years of Internet use on political engagement is negative. The negative effect contradicts other analyses of the same data set (Quan-Haase et al., 2002; Wellman et al., 2001) and other studies in the meta-analysis that suggest the effect is positive (Moy et al., 2005). The relationship of Internet experience and engagement merits further investigation, as none of the studies are based on random samples of the national population.

Gibson et al.'s (2000) second negative effect is based on the relationship between online socializing and political engagement. The online socializing index includes participation in online political discussions, participation in listservs, and other measures. The negative effect is contradicted by other analyses of the NGS survey. Wellman et al. (2001) find that online political discussion is positively and significantly related to offline political activity. Other data sources also contradict the negative effect documented by Gibson et al. (2000). These studies find that online political discussion has a significant, positive effect on offline political participation (Hwang, Schmierbach, Paek, Zuniga, & Shah, 2006; Mossberger et al., 2008; Nah, Veenstra, & Shah, 2006; Price, Goldthwaite, Cappella, & Romantan, 2002; Shah et al., 2005; Shah, Cho, et al., 2007). Gibson et al. (2000) also include listservs in the online socializing index, but this indicator likely does not explain the negative effects. Several studies include this measure in their index and find positive effects on engagement (Moy et al., 2005; Norris & Jones, 1998). In sum, Gibson et al.'s (2000) two significant, negative effects seem contradictory to other NGS studies and other studies of Internet use and engagement. As an aggregate, the meta-data provide little evidence of a significant, negative effect of Internet use on engagement.

When all effects are averaged, the estimated effect is .07 with a standard deviation of .098 (22 studies, 85 effects). The average effect size is within one standard deviation of zero. This estimate is consistent (within .017) however the average effect is calculated (i.e., study level or data source level). ⁷ This small effect raises questions about whether the effect is substantial. However, this average effect calculation does not take into account variations in the measurement of Internet use and year of data collection, which alter the observed effects (see Table 2).

Table 2 Aggregate findings by study characteristic (bivariate analysis)

	%of significant effects	% of positive effects	Mean effect(SD)
Use online news in measure of Internet use	56	97∗∗∗	.13∗∗∗ (.103)
	n = 68	n = 58	n = 29
Years of data collection
1995–1997	13∗∗	91	.04 ^# (.055)
	n = 23	n = 21	n = 8
1998–1999	61∗∗	73	.04 ^# (.058)
	n = 51	n = 48	n = 35
2000–2001	53∗∗	91	.07 ^# (.117)
	n = 34	n = 32	n = 11
2002–2003	46∗∗	82	.11 ^# (.130)
	n = 50	n = 44	n = 27
2004–2005	63∗∗	100	.09 ^# (.044)
	n = 8	n = 8	n = 4
Control for political interest	35∗	90 ^#	.04 (.075)
	n = 65	n = 60	n = 12
Control for media use	53 ^#	83	.07 (.106)
	n = 118	n = 107	n = 67
RDD sample	37∗∗	87	.05 (.066)
	n = 78	n = 78	n = 27
Web survey	73∗∗∗	82	.08 (.094)
	n = 40	n = 33	n = 27
Total	48	77	.07 (.098)
	n = 166	n = 166	n = 85
Note. The 1997 data are not used in computing the average effect, because the analysis does not use OLS and does not report standardized coefficients.
^# p ≤ .10; ∗p ≤ .05; ∗∗p ≤ .01; ∗∗∗p ≤ .001. Two-tail test.

When the Internet use measure includes online news or information about public affairs or campaigns, the effects are more likely to be positive (p ≤ .001) and larger (p ≤ .001), but not more likely to be statistically significant (Table 2; p = .115). When this type of use is included in the measure of Internet use, the average effect size is larger than the average effect size when other measures are used (.13 versus .04 for all other studies; p ≤ .001). While online news may produce larger effects, the degree to which using online news explains variations in political engagement is small. Some studies estimate the explained variance at less than 1% (Bimber, 2003; Hardy & Scheufele, 2005; Kenski & Stroud, 2006; Nisbet & Scheufele, 2004; Scheufele & Nisbet, 2002), less than 2% (Shah et al., 2001b), or about 3% (Kwak et al., 2006; Moy et al., 2005). These findings raise questions about whether Internet use has any substantial effect on civic and political engagement.

The relationship of Internet use and engagement may change across time. Other studies have examined changing effects across time, comparing the effects for years with presidential elections and those years without presidential elections (Mossberger et al., 2008; Tolbert & McNeal, 2003). They compare the effects of using online news (campaign information) on voting, using data from 1998 and 2002 compared to 1996, 2000, and 2004. They conclude that the effects are not significant in 1998 and 2002, but are significant in years with presidential elections. In contrast, the meta-data do not indicate that there are differences in the proportion of significant effects, the proportion of positive effects, or changes in the average effect for years with presidential elections and those years without presidential elections. ⁸

The analysis suggests that the effects of Internet use on engagement are increasing across time, but the change is not monotonic. The analysis of how the effect size changes across time must be done cautiously, since the number of effects reported in each year changes dramatically (Table 2). In addition, because a single study tends to report data from a single year, the effect of study design has a large impact on the yearly averages. To help address these concerns, the dates were collapsed into multiple years (Table 2). The proportion of significant effects (p ≤ .01) and the average size of the effect (p ≤ .10) increase across time, but the changes are not in a linear direction (Table 2). The year 2000 seems to be a key transitional year for average effect sizes. The proportion of significant effects is highest in 1998/1999 and 2004/2005. However, the findings from 1998/1999 may be explained by the sampling approach of the two predominant data sources in this time period.

Discussion

The relationship of Internet use and engagement differs depending on the causal modeling approach and characteristics of the data source (e.g., sampling). Three caveats are necessary when evaluating this body of research. The first caveat relates to the inclusion of political interest in the causal model, and the second caveat relates to causal direction. The third caveat is about the number of self-selected samples used in this research area. The presence of political interest in the model, the causal direction assumed in the model, and the use of self-selected samples change the findings related to the number of significant effects observed in the meta-data.

Mediated Effects or Spuriousness

When political interest and Internet use are combined in a model predicting engagement, studies tend to find that the effect of Internet use is not statistically significant (Table 2; p ≤ .05). Only 35% of studies that control for political interest report statistically significant effects. These studies that control for political interest are also more likely to report positive effects (Table 2; p ≤ .10); this pattern may be attributed to these studies' tendencies to use online news as a measure of Internet use. To illustrate these findings, consider Bimber's (2001) and Norris's (2000) analyses of ANES 1998 data. Both researchers examine this data set using the same independent and dependent variables. Both researchers report positive effects. Bimber includes political interest in his model and finds that the effect of online campaign information is weak and not statistically significant. Norris does not include political interest and finds a significant effect of online campaign information on engagement.

Other studies cannot be as precisely compared, because these studies vary other characteristics that may explain the observed effects of Internet use on engagement, such as year of data collection and other statistical controls (other media uses). The studies tend to affirm that the effect of Internet use on engagement is not significant when political interest is controlled for in the model. For example, Krueger and Bimber examine the effects of political interest and Internet use on engagement without controlling for other media uses (Best & Krueger, 2005; Bimber, 2001, 2003; Krueger, 2002, 2006). ⁹ Using data from 1998, 2000, and 2003, these researchers tend to find that the effect of Internet use on engagement is not significant when political interest is controlled for in the model.

Another group of studies also examine political interest, online news, other media uses, and engagement. These studies find small and/or insignificant effects of Internet use on engagement when political interest is controlled for in the model (e.g., Mossberger et al.'s, 2008, analysis of 2002 PEW data; see also Jennings & Zeitner, 2003; Kenski & Stroud, 2006; Kwak et al., 2004; Xenos & Moy, 2007). These studies provide data from 1997, 2000, 2002, and 2004. Of the 17 effects tested in these five studies, only four effects (24%) of Internet use on engagement are statistically significant.

On the other hand, Bimber's (2003) analysis of ANES 2000 data reveals small but significant effects of Internet use on engagement when political interest is controlled for (p < .05 and p > .01). These positive, significant effects are reaffirmed in other studies using ANES 2000 regardless of whether political interest or other media uses are controlled for in the model (Mossberger et al., 2008; Nisbet & Scheufele, 2004; Norris, 2003; Tolbert & McNeal, 2003). In addition, two other studies find significant effects of Internet use on engagement when political interest is controlled for in the model (Hardy & Scheufele, 2005; Mossberger et al., 2008). These studies use random, national sample data from a 2002 survey and a 2004 PEW survey. Based on these studies, the role of political interest in influencing the observed effect of Internet use on engagement may be changing across time.

I explain these patterns of findings as mediated effects. Because the Internet use variable, rather than the political interest variable, tends to become nonsignificant, I identify political interest as the mediator in the relationship between Internet use and engagement (see Figure 1, paths d and e). ¹⁰ However, these patterns of findings may also be indicative of spuriousness (see Figure 1, paths c and f). If political interest causes both Internet use and engagement, the observed correlation between Internet use and engagement could be explained by this common cause. The meta-data are insufficient to evaluate the plausibility of mediated or spurious effects, because existing research has not established the causal direction of Internet use and political interest. Some researchers assess whether use of online news Web sites predicts political interest (Lupia & Philpot, 2005; Mossberger et al., 2008), while others treat political interest as a predictor of Internet use (Best & Krueger, 2005; Bimber, 1998; Jennings & Zeitner, 2003; Johnson & Kaye, 2003; Kwak et al., 2004; Shah & Scheufele, 2006; Xenos & Moy, 2007). Most of these studies rely on contemporaneous measures, which cannot establish the causal direction. Analyses of panel data surveys assume the causal direction of effects rather than testing the different flow of effects. The relationship among these three variables (i.e., political interest, Internet use, and engagement) is a critical area for further research.

Causal Process

The second caveat around the body of research relates to causal direction. Research tends to treat Internet use as a predictor of engagement, assuming that Internet use can affect levels of engagement. However, a handful of studies test the alternative assertion—those who are more politically engaged use the Internet more (see Figure 1, path c). This proposition is in line with Norris's (2000) virtuous circle. The findings from this treatment of the causal process are contradictory, but tend to suggest that engagement does not have a significant effect on Internet use. Five studies with 14 effects treat causality as running from engagement to Internet use (Jennings & Zeitner, 2003; Katz & Rice, 2002; Krueger, 2006; Norris & Jones, 1998; Wellman et al., 2001). Of these 14 effects, only 21% are statistically significant. The studies tend to find a positive effect of engagement on Internet use (Krueger, 2006; Price et al., 2002; Norris & Jones, 1998; Katz & Rice, 2002; Jennings & Zeitner, 2003), with the exception of Shah, Schmierbach, Hawkins, Espino, and Donavan (2002).

All of these studies, except Jennings and Zeitner (2003), involve contemporaneous measures of Internet use and engagement and cannot truly assess causal direction. As such, Jennings and Zeitner's (2003) findings offer the most conclusive evidence that when the effects are modeled as engagement causing Internet use, the tendency is to find that the effects are not significant. Unfortunately, this study also controls for political interest, which may also explain the lack of significant effects between Internet use and engagement.

These differing claims about the causal process suggest that reciprocal effects should be considered. Shah et al. (2002) examine the reciprocal relationship between time spent online and civic engagement. Their conclusion is that time spent online leads to engagement, rather than vice versa. The standardized coefficients are .08 for time spent online leading to engagement and –.08 for engagement leading to time spent online. The positive coefficient is statistically significant, and the negative coefficient is not. This research raises questions about whether Internet use has a substantial effect on political engagement, but the study also affirms the tendency of finding no significant effects when causality is tested as engagement leading to Internet use.

Experiments offer the best hope of untangling the direction of effects and the nature of the causal relationship. Price et al. (2002) completed an experiment related to online discussion forums and found that participation in these forums increased the likelihood of voting in the next election, controlling for habitual voting. Habitual voting was positively related to participating in the online forums (Price & Cappella, 2002). Their findings suggest a two-way causal process, but they did not explicitly test a reciprocal effects model.

Self-Selected Sample Surveys

Twenty-one of the 38 studies (78 of the 166 effects) use RDD samples. RDD sample studies produce different findings than other samples, and they are less likely to find significant effects than other studies (Table 2; p ≤ .01). If RDD samples are believed to produce higher quality results, then the inclusion of other studies may overestimate the likelihood of a significant effect of Internet use on engagement. This section highlights data quality issues associated with two types of studies: Web surveys with respondents recruited from Websites and mail surveys with respondents recruited from a list of volunteers.

Eight of the 38 studies (40 of the 166 effects) use data from self-selected Web surveys. In general, the researchers using Web surveys pay little attention to how their method of recruitment and survey administration influence the observed effects. For example, the National Geographic Society Survey was posted on the National Geographic Society Web site, with some respondents recruited through libraries, workplaces, schools, discussion groups, and other Web sites. The National Geographic Society Web site is used to recruit participants and host the survey. The measure of Internet use is use of online magazines (all NGS survey analyses except Quan-Haase et al., 2002). ¹¹ In another example, political Web sites are used to recruit participants, with the independent variable being use of these types of Web sites (Johnson & Kaye, 2003). ¹² Finally, another example is Hwang et al. (2006) and Nah et al. (2006), who use the same Web survey data. Both sets of researchers include online political discussion as a variable in their model, and their survey respondents are recruited from listservs, weblogs, and online discussion boards. In sum, these Web surveys are problematic because the measures of Internet use reflect how respondents were recruited or reflect the survey administration mechanism. As a result, the distribution of the independent and dependent variables may be skewed.

Web surveys, in general, do not consistently report higher or lower effect sizes or positive versus negative effects. They do have a greater tendency to conclude that effects are significant compared to other surveys (Table 2; p ≤ .001). The National Geographic Society survey exemplifies this pattern but does not totally account for it, as smaller sample surveys also demonstrate this tendency. Gibson et al. (2000), Quan-Haase et al. (2002), Weber et al. (2003), and Wellman et al. (2001) all use this data set. The NGS survey has over 20,000 respondents. This survey is the source of 28 effects, and 22 of the effects are significant (79%), which is a much higher percentage than the meta-data in general (p ≤ .001). The average effect from this data source is smaller than other studies, but this difference is not statistically significant. For example, Quan-Haase et al. (2002) find nine positive, statistically significant effects of different measures of Internet use on engagement. All of these standardized effects are .10 or lower.

The DDB Life Style Study accounts for 24 effects in seven studies (Shah, Cho, et al., 2007; Shah, McLeod, et al., 2007; Shah & Scheufele, 2006; Shah et al., 2005; Shah et al., 2002; Shah, Kwak, & Holbert, 2001; Shah, McLeod, & Yoon, 2001). Comparing this data source to other data sources is difficult, because the data were collected in multiple years, which could explain why the findings produced differ from other data sources. In addition, evaluating the findings from this data source is difficult, because 6 of the 24 coefficients derived from this source do not have their direction reported (positive or negative). These missing data for the meta-analysis make it difficult to accurately assess patterns of effects in this data source. The data source produces similar results as other sources in terms of the likelihood of finding a significant effect. The studies using 1998 and 1999 data are less likely (54%) to find positive effects than other studies. The studies using 2002, 2004, and 2005 data are more likely (100%) to find positive effects than other studies (p ≤ .10).

This data source raises concerns because of its sample design. People on commercial lists are contacted and asked to volunteer to participate in various surveys (see Putnam, 2000, for a description of the composition of the lists; no details are available about the level of coverage of these commercial lists). About 10% or less of people agree to receive surveys, and of those volunteers about 70% to 80% return a completed survey (Putnam, 2000). These layers of recruitment raise questions about the sample and response bias. The sample becomes a random sample of a small group of volunteers. While the demographic representativeness of the survey participants has been evaluated, the sample may be biased in its representation on other key variables such as media usage (see Putnam, 2000, pp. 420–422). Likewise, the sample may be biased in terms of Internet use, which may produce findings that are different from other data sources. In sum, studies using self-selected surveys tend to produce estimates of the effects of Internet use and engagement that differ from RDD sample surveys.

Conclusion

In this analysis of 38 studies and 166 effects testing the relationship between Internet use and political engagement, the meta-data establish that there is little evidence to support the argument that Internet use is contributing to civic decline. The findings suggest that the effect of Internet use on engagement is positive. However, the question remains: Are these effects substantial? The average positive effect is small in size.

Two factors decrease the likelihood of finding statistically significant effects. These factors are the inclusion of political interest in the causal model and specifying the causal direction as engagement causing Internet use. These two factors merit further research to assess what the true relationship is between Internet use and engagement. In addition, further studies should assess how the relationship may have changed across time. Studies completed prior to 1998 may have been too quick to assume null effects, as the effect of Internet use on engagement had yet to manifest itself. Data collected in 2000 and beyond tend to produce larger effect sizes than data collected prior to this period. The meta-data suggest increasing effects across time, but the change is not monotonic. The research design of studies may account for the nonlinearities. If the effect is increasing across time, in the long run Internet use, particularly use of online news, may produce a substantial effect on engagement.

Measuring Internet use as online news tends to increase the likelihood of finding a positive and larger effect of Internet use on engagement. This finding suggests that increased access to a large, diverse set of political information may help reinvigorate civic life. In other words, the Internet may reduce the costs of participation (time, effort) by increasing the availability of information. Further research should more fully explore the Internet's varied effects, considering other types of Internet use and specific civic and political activities. These more nuanced measures and findings would help advance theory about how Internet use affects engagement.

This meta-analysis focused on bivariate relationships between the meta-data and the features of the research design or causal model (Table 2). This analysis approach was necessary because of the small sample of effects, but makes it difficult to isolate the influence of different features on the observed effects. This bivariate analysis reveals several findings that should be tested in further research. First, the ANES, or some other data source, could be used to examine whether the effects of Internet use on engagement are increasing across time. This analysis would need to be sensitive to the role of political interest in shaping the observed effect between Internet use and engagement. Second, further research should explore a two-way causal process, because the significance of the relationship seems to differ depending on whether the relationship is modeled as Internet use causing engagement or vice versa. In sum, this meta-analysis suggests that the effects of Internet use on engagement are positive, but does not establish that these effects are substantial.

Notes

1. While the intent was to be inclusive, I did draw a line between civic and political engagement and what is more often referred to as social capital or social engagement. Many studies consider the impact of Internet use on engagement alongside measures of social capital (Bimber, 2001, 2003; Gibson, Howard, & Ward, 2000; Jennings & Zeitner, 2003; Johnson & Kaye, 2003; Katz & Rice, 2002; Kwak, Skoric, Williams, & Poor, 2004; Moy, Manosevitch, Stamm, & Dunsmore, 2005; Quan-Haase, Wellman, Witte, & Hampton, 2002; Shah, Kwak, & Holbert, 2001; Shah, McLeod, & Yoon, 2001; Weber et al., 2003). These studies were included in the analysis, because they examined civic or political engagement. However, studies (or components of studies) that examine social capital as the dependent variable were not included in the meta-data. The distinction is most apparent for organizational memberships. Volunteering for an organization is a form of engagement, while membership is a form of social capital (Paxton, 1999). The focus of this meta-analysis is on political and civic activities, rather than the effect of Internet use on social capital (see other works, such as Shklovski, Kiesler, & Kraut, 2006).

2. This definition does not include measures related to intentions or willingness to perform various political or civic activities. The reason for this exclusion is that when intentions or willingness to perform political activities are used as measures, the estimated effects of Internet use tend to be higher than when actual behavior is assessed. For example, Johnson and Kaye (2003) evaluate both intention to vote and voting; they find that Internet use has strong (and sometimes statistically significant) effects on intentions to vote, but no significant effects on actual voting behavior. Likewise, Scheufele and Nisbet (2002) look at willingness to participate in public forums and actual political behaviors and find higher estimated effects on the willingness index. In another example, Kwak, Poor, and Skoric (2006) report stronger effects of online news use on willingness to engage in international political activities, compared to participation in actual domestic political activities. The choice to exclude behavioral intentions results in excluding several studies, such as Kaye and Johnson (2002) and Lupia and Philpot (2005). However, this exclusion was necessary, because the measurement of political behavior (i.e., actual versus willingness) leads to different effects, which diminish the reliability of the average effect size.

3. Assessing patterns of statistical significance may be perceived as problematic given that statistical significance assumes random selection, reflects the size of the sample, and duplicates findings that can be derived from calculating the magnitude of the effect. However, this focus is justified given the hypotheses being tested as well as the data limitations in this body of research. Many studies do not report coefficients that could be used in calculating the average effect size. As such, the findings regarding statistical significance provide a unique source of meta-data findings.

4. Best and Krueger (2005) use the same data set as Krueger (2006) and find a negative effect. This negative effect is consistent with Kwak et al. (2004) but opposite of Krueger (2006).

5. This finding is based on the analysis of meta-data and personal communication with Ramona McNeal regarding Mossberger, Tolbert, and McNeal (2008) and Tolbert and McNeal (2003) on March 25, 2008.

6. This study accounts for 18 effects. The average effect from this study is higher than all other studies (p ≤ .05), but the study does not differ significantly from other studies in the proportion of significant effects or proportion of positive effects.

7. The average effect is calculated based on all studies using OLS and reporting standardized coefficients (or unstandardized coefficients and standard deviations, such as Kraut et al., 2002). The median is .06. The mean effect was also calculated by averaging study-level effects into one coefficient and then computing the mean (M = .08, SD = .075, median = .05) and by averaging data source–level effects into one coefficient and then computing the mean (M = .09, SD = .078, median = .05).

8. This relationship is difficult to explore because of the correlation among measures of Internet use (e.g., online news), measures of engagement (i.e., focus on voting), and characteristics of studies done in years with presidential elections (RDD). These different types of measures and characteristics of studies may simultaneously suppress and exaggerate the average effect. To control for the effects of different types of measures and characteristics of studies, I examined the effects using ANES 1996, 1998, 2000, and 2004. Of the 29 effects across these years of data collection, the likelihood of a significant effect, positive effect, and average effect size do not differ for election years versus years without a presidential election (p > .10). However, this is a small set of effects. Additional research is required.

9. When media uses are included in the model, the effect is more likely to be reported as statistically significant, compared to studies that do not include media uses in the model (p ≤ .10; Table 2). This finding may reflect the year of data collection; studies that control for media use tend to be more recent in their data collection.

10. The effect of Internet use on engagement may also be mediated by political knowledge or both political interest and knowledge, but these relationships have been given less attention. Scheufele and Nisbet (2002) find that there is no statistically significant effect of political knowledge or Internet use on engagement. Flipping the causal direction, Jennings and Zeitner (2003) tend to find no relationship between engagement and Internet use in a model that controls for political interest and knowledge. Norris and Jones (1998) also flip the causal direction and find no relationship between engagement and Internet use when political knowledge is controlled. In contrast, Mossberger et al. (2008) examine ANES 2000, controlling for political interest and knowledge, and find a significant effect of Internet use and engagement. Again, ANES 2000 produces findings that are contradictory to meta-data patterns.

11. Witte, Amoroso, and Howard (2000) attempt to establish the representativeness of the American sample that participated in the National Geographic Society survey, but focus on demographic variables instead of comparing how variation on key dependent variables differs from other surveys. Weber et al. (2003) try to compare frequencies on the dependent variables, but outline the difficulty given the variation in question wording. Their comparison suggests that those who responded to the National Geographic Society survey report higher levels of engagement, which Weber et al. explain in terms of the sample being more highly educated than the general population.

12. Johnson and Kaye (2003) note that their survey respondents may underrepresent women and racial minorities. They note that the intent of their sample design is to attract politically interested Web users, which is problematic in that they then try to examine political interest as a dependent variable (Johnson & Kaye, 2003). Sampling on the dependent variable, for example political interest (or in Kaye & Johnson, 2002, on Internet use), may bias the results toward finding no effects, when an effect does exist (Geddes, 1990; King, Keohane, & Verba, 1994).

Appendix: List of Meta-Analysis Sources

Table

Author(s)	Source
Best & Krueger, 2005	Table 1 (p. 192) of original paper
Bimber, 2001	Table 2 (p. 62) and Table 3 (p. 63) of original paper
Bimber, 2003	Table 5.6 (p. 222) of book
Gibson et al., 2000	Table 4 (no page numbers identified)
Hardy & Scheufele, 2005	Table 1 (p. 77) and Table 2 (p. 79) of original paper
Hwang et al., 2006	Figure 1 (p. 472) of original paper
Jennings & Zeitner, 2003	Table 4 (p. 324) and Table 5 (p. 325) of original paper
Johnson & Kaye, 2003	Table 3 (p. 24) and pages 24, 25 (no table) of original paper
Katz & Rice, 2002	Table 7.3 (p. 141) and Table 7.4 (p. 143) of book
Katz & Rice, 2002	Table 7.7 (p. 147) of book
Kenski & Stroud, 2006	Table 3 (p. 186) of original paper
Kim et al., 2004	Table 5 (p. 625) of original paper
Kraut et al., 2002	Table 4 (p. 63) or original paper
Krueger, 2002	Table 2 (p. 489) of original paper
Krueger, 2006	Table 1 (p. 768) of original paper
Kwak et al., 2006	Table 4 (p. 205) of original paper
Kwak et al., 2004	Table 2 (p. 434) and Table 3 (p. 436) of original paper
McCluskey et al., 2004	Table 2 (p. 447) of original paper
Mossberger et al., 2008	Table 3.A.1 (pp. 171–172) of book
Mossberger et al., 2008	Table 4.A.1 (pp. 177–178), Table 4.A.2 (pp. 179–180), and Table 4.A.3 (pp. 181–182) of book
Moy et al., 2005	Table 2 (p. 578) of original paper
Nah et al., 2006	Figure 2 (p. 239) of original paper
Nisbet & Scheufele, 2004	Table 2 (p. 888) of original paper
Norris, 2000	Table 13.5 (p. 291) of book
Norris, 2003	Table 2 (p. 13) of online version of article
Norris & Jones, 1998	Table 2 (p. 3) of original paper
Price et al., 2002	Table 2 (p. 40) of original paper
Quan-Haase et al., 2002	Table 10.4 (p. 311) of book
Scheufele & Nisbet, 2002	Table 3 (p. 67) of original paper
Shah et al., 2005	Figure 2 (p. 545) of original paper
Shah, Cho et al., 2007	Figure 2 (p. 690) and Figure 3 (p. 691) of original paper
Shah, Kwak, & Holbert, 2001	Table 2 (p. 150) of original paper
Shah, McLeod et al., 2007	Table 1 (p. 227) and Table 2 (p. 229) of original paper
Shah, McLeod, & Yoon, 2001	Table 4 (p. 487) of original paper
Shah & Scheufele, 2006	Figure 1 (p. 10) of original paper
Shah et al., 2002	Table 5 (p. 977) of original paper
Tolbert & McNeal, 2003	Table 1 (p. 180) and Table 3 (p. 182) of original paper
Weber et al., 2003	Table 4 (p. 38) of original paper
Wellman et al., 2001	Table 4 (p. 446) and Table 5 (p. 447) of original paper
Xenos & Moy, 2007	Table 2 (p. 713) of original paper