Political identity moderates the effect of watchful eyes on voter mobilization: A reply to Matland and Murray (2019)

ABSTRACT

Matland and Murray (2019) reanalyze three of their previous field experiments and fail to reproduce the finding reported in Panagopoulos and van der Linden (2016) that political identity moderates the watchful eye effect on voter mobilization in elections. We highlight several concerns with their empirical approach, including lack of power and between-study differences, that lead us to conclude that the authors offer little evidence against partisan heterogeneity. By contrast, closer inspection and additional analyses of the data reported in Panagopoulos and van der Linden (2016) only reinforce our original conclusion that partisanship moderates voter responsiveness to watchful eyes. Specifically, Republicans appear to be more susceptible to watchful eyes compared to Democrats in the context of voting in elections.

KEYWORDS:

• watchful eyes
• implicit social cues
• political identity
• voter turnout

A burgeoning literature has found that watchful eyes can encourage prosocial behavior and human cooperation in classic social dilemmas (Bateson, Nettle, & Roberts, 2006; Burnham & Hare, 2007; Ernest-Jones, Nettle, & Bateson, 2011), including voting in elections (Panagopoulos, 2014a, 2014b). Recent meta-analyses have offered substantial disagreement on the replicability of the watchful eye effect, ranging from no discernible effect (Northover, Pedersen, Cohen, & Andrews, 2017), to a small positive effect (Bradley, Lawrence, & Ferguson, 2018) to quite large effects in realistic field settings (Kelsey, Vaish, & Grossmann, 2018). Equally, meta-analyses show large robust effects of watchful eyes on reducing antisocial behavior, even when compared to actual observability cues (Dear, Dutton, & Fox, 2019). In an attempt to explain such a significant degree of heterogeneity between studies, scholars have started to explore the underlying psychological processes and moderators that could explain why implicit social cues enhance human cooperation, including the direction of the gaze (Manesi, van Lange, & Pollet, 2016), emotional responses to gaze (Panagopoulos & van der Linden, 2017), and of current interest, the important role of political identity (Panagopoulos & van der Linden, 2016). Based on a large literature, we hypothesized that because conservatives have a stronger desire to share reality with like-minded others and value conformity and obedience more than liberals (Jost, Nosek, & Gosling, 2008; Jost, van der Linden, Panagopoulos, & Hardin, 2018), this finding would extend to susceptibility to implicit social cues, such as watchful eyes. Our findings confirmed this hypothesis. Matland and Murray (2019) conducted a series of five replications of (Panagopoulos, 2014a, 2014b) to examine the influence of watchful eyes on voter mobilization and found no statistically significant main effect of their watchful eye intervention. The authors returned to three of their field studies for which political affiliation could be deduced, and, in contrast to Panagopoulos and van der Linden (2016), they report no significant partisan heterogeneity in the watchful eye effect. The authors also raise several other criticisms and conclude that, overall, they find no evidence for partisan differences. Below we respond to these critiques, discuss several shortcomings we perceive in Matland and Murray (2019), and undertake additional analyses of data reported in Panagopoulos and van der Linden (2016) to show that our original conclusion that partisanship moderates responsiveness to watchful eyes is robust. Specifically, we contend that Republicans appear more susceptible to watchful eyes than Democrats in the context of voting in elections.

We begin by acknowledging the possibility that the null results reported in Matland and Murray (2019) mask meaningful treatment effect heterogeneity, whereby different treatments have differential causal effects on each unit or subpopulation (Green & Kern, 2012; Imai & Ratkovic, 2013). As Green and Kern (2012, p. 491) state, the informativeness of the ATE as a summary of the distribution of treatment effects depends on the extent and form of treatment effect heterogeneity in any given experiment (see also Imai & Strauss, 2011). It is therefore not implausible in the watchful eyes experiments in question that there exists a cross-over pattern, with a positive effect for one group (e.g. Republicans) and a negative effect for the other group (e.g. Democrats). In other words, the lack of a main effect in Matland and Murray (2019) could be explained by a positive effect of one group canceling out the negative effect of the other group.

Commonly, however, null results do not mask treatment heterogeneity and, instead, are obtained when treatments fail to exert their anticipated effects or when experimental protocols or implementation are ineffective. In such scenarios, significant treatment-by-covariate interactions would be unlikely. Indeed, the lack of a main effect in Matland and Murray (2019) is not explained by a potential cross-over interaction. Furthermore, we cannot rule out the possibility that the null results reported in Matland and Murray (2019) reflect experimental failures attributable to any number of sources and, in particular, key features that differed from the protocols implemented in Panagopoulos (2014a, 2014b) and subsequently reanalyzed in Panagopoulos and van der Linden (2016).

In addition, without more detailed information about statistical power, it is difficult to evaluate the null findings reported in Matland and Murray (2019) in proper context. This is especially true in the absence of strong theory that would predict null findings. Indeed, Matland and Murray (2019) agree with our assessment that the literature in political ideology has shown robust psychological differences between conservatives and liberals. Yet, the authors provide no theoretical explanation that would support an a priori expectation to observe such differences. Thus, assuming Matland and Murray (2019) set out to detect a difference, more detail is needed on statistical power. For example, in related research, Weinschenk, Panagopoulos, Drabot, and van der Linden (2018) show that, with sufficient statistical power to detect even small gender differences in susceptibility to social norm messages on voter mobilization, no consistent gender differences were actually observed.

Although Matland and Murray (2019) note – and swiftly dismiss – the importance of statistical power when presenting their null findings, the watchful eye effect is known to be relatively small (Bradley et al., 2018), and Matland and Murray (2019) provide no evidence that their design was sufficiently statistically powered to detect small effect-sizes, let alone small differences between Republicans and Democrats. In short, we believe statistical power, which is a function of both the size of an effect as well as the size of the sample used to detect it, is crucial when seeking to estimate the likelihood that a study will fail to reject a false null hypothesis (or making a Type II error). Accordingly, we reject the notion that statistical power is ever “immaterial” under any circumstances (Murray, 2019). We are also puzzled by Murray’s (2019) unrealistic claims about “unlimited power.” To illustrate, using G*Power (Faul, Erdfelder, Lang, & Buchner, 2007), we conducted a sensitivity analysis for a linear regression that tests for the difference between slopes. Taking the El Paso site as an example, with ɑ = 0.05, and sample sizes of 6,192 and 20,671, respectively, to maintain 95% power, the minimum required effect size between Republicans and Democrats is 0.01. The actual observed difference of 0.001 (Matland & Murray, 2019, Table 2, p.6) is thus far below this threshold and corresponds to only 10% post-hoc observed power. While perhaps less of a concern for the other sites, which have larger samples, for at least El Paso, we cannot be confident that the reported difference is a true null effect.

Moreover, even for the main effect of watchful eyes, a sensitivity test for the Midland sample (N = 18,170) reveals that with just 80% power, the authors’ sample can detect a small effect in a linear bivariate regression (β = 0.018). Yet, the actual coefficient for the observed effect of watchful eyes (β = 0.014, Table 2) still falls below this threshold, calling into the question whether their studies were sufficiently powered from the start. Indeed, the 95% confidence interval for the expected effect of the watchful eyes interventions, given the pooled, estimated average treatment effect reported in Panagopoulos (2014b), relative to the control conditions, implies the effect likely ranges from about 0.33 to 2.17 percentage points. If the effect is, in fact, as small as the lower end of this range, a sample of about 570,000 is required to achieve 80% power, assuming all subjects assigned to be treated are contacted and half are assigned to treatment and the other half to control conditions; the estimated average treatment effect of 1.25 percentage points reported in Panagopoulos (2014b) implies a sample of at least 39,000 is required for 80% power, if half of the sample is treated and all subjects assigned to be treated are contacted. The overall sample sizes of the five replications Matland and Murray (2019) conducted imply power for a one-sided, 5% test ranged from .40 to .63, given the fractions assigned to be treated and assuming all subjects assigned to be exposed to watchful eyes were successfully contacted.

We concede that pooling across studies would yield sufficient power to detect an effect as small as about 0.9 percentage points, but even the pooled sample lacks adequate power to detect smaller effects.¹ Furthermore, the power calculations suggest the authors likely lacked sufficient power in their individual studies to detect main effects along the lines one can expect given the estimated average treatment effects reported in Panagopoulos (2014b), unless the true magnitude of the effect was [much] closer to the upper range implied in Panagopoulos (2014b).

Another criticism that Matland and Murray (2019) offer is that the difference in coefficients between Republicans and Democrats reported in Panagopoulos and van der Linden (2016) was not statistically significant. Importantly, we simply noted that the treatment effect itself (compared to the control group) was significant for Republican respondents across both the Key West, FL (3.0%) and Lexington, KY (4.4%) sites but not for Democrats (0.6% and 1.4%, respectively). We concede that the wording of our initial hypothesis (‘a stronger effect of eyespot images on voter mobilization for Republicans compared to Democrats’) lends itself to a more rigorous test of interaction or equivalence in coefficients. To increase power and synthesize our findings, we pooled the two studies we analyzed in Panagopoulos and van der Linden (2016).

Contrary to Matland and Murray's (2019) assertions, a Chow test of equivalence between Democrats and Republicans on the pooled dataset across sites using SUEST (seemingly unrelated regressions) returns a significant difference (β_rep = 0.039, SE = 0.015 vs. β_dem = 0.01, SE = 0.010, β_diff = 0.029, χ² [2] = 19.34, p = 0.001). Another way to investigate the same question is to simply test for an interaction effect. Including a treatment (eyes) and party ID (republican) interaction term in the regression model on the pooled data results in a significant interaction term (β = 0.039, SE = 0.02, p = 0.05, two-tailed, Table 1), such that being assigned to the watchful eye condition has a stronger effect if the participant is Republican (vs. Democrat). The bar graph (Figure 1) illustrates our original point: for Republicans, the watchful eye effect is significant (Republican panel) but not for Democrats (Democrat panel). In contrast to Matland and Murray (2019), pooling the data also highlights that the difference between treatment and control for Republicans (0.039) is significantly larger (p = 0.001) than the difference in coefficients for Democrats (0.01).

Figure 1. Marginal linear predictions from interaction term on pooled dataset. Note: Error bars indicate 95% confidence intervals.

Table 1. Linear regression on pooled dataset.

Pr(Voted)	β
Partisan identity (Republican)	−0.028***
	(0.003)
Treatment (Eyes)	0.006
	(0.012)
Interaction (Republican*Eyes)	0.038*
	(0.020)
Site (Key West)	0.195***
	(0.004)
N	72,669
F	449.07
RMSE	0.409

***p < 0.001, **p < 0.01, *p = 0.05 (two-tailed). Coefficients are unstandardized derived from OLS linear regression. Standard errors in parentheses.

In addition, in light of recent calls to favor effect-sizes over statistical significance (Amrhein, Greenland, & McShane, 2019), we interpret the average magnitude of the difference (about 3.0 percentage points) in voting behavior between Republicans and Democrats in response to watchful eyes as not only statistically significant but also substantively meaningful, especially when scaled across individuals (Funder & Ozer, 2019) and in light of relatively low baseline voting rates in local elections.

Further, we recognize that there are important between-study differences that could explain divergence in our results. In contrast to the studies we analyze in Panagopoulos and van der Linden (2016), Matland and Murray (2019) used computer-generated eyes, the method of determining partisanship differed, and there were important sampling differences as well (e.g. single-voter vs. multiple-voter households).

Although Matland and Murray (2019) are rather dismissive of the importance of any differences between our samples, treatments, and methodologies, the observed between-study heterogeneity is rather substantial, and such discrepancies are known sources of inconsistencies in replication efforts (McShane, Tackett, Bockenholt, & Gelman, 2018). In fact, to illustrate the importance of these differences, consider that, in our studies, partisanship was known because subjects’ party registrations were recorded and available in official voter files obtained from the experimental jurisdictions. We leveraged these designations without modification or any further assumptions. By contrast, Matland and Murray (2019) adopted a very different approach to determine subjects’ partisanship. The authors used voting in past primaries to denote partisanship. Specifically, only subjects who had voted exclusively in Democratic (Republican) primaries were categorized as Democrats (Republicans), respectively, while subjects who had not voted in any party primaries were categorized as independents. We have several concerns about the reliability of this approach, including that it relies on voting in primaries that routinely attract far fewer voters than general elections, and that it fails to take into account that partisans often cross party lines to vote in primaries, especially in open primary states in which the authors conducted their tests. In Matland and Murray (2019), such voters were excluded from the analyses. Notwithstanding these concerns, our main contention is that this approach was quite different from the one adopted in Panagopoulos and van der Linden (2016) and could have conceivably classified partisan subgroups of voters in a way that was incomparable across the two sets of studies. This seems especially relevant given that the focus of the studies revolves around political identity.

Still, other noteworthy differences across the sets of studies exist: Unlike Panagopoulos (2014a, 2014b), Matland and Murray (2019) use stylized rather than realistic eyes (as in Panagopoulos, 2014a, 2014b) in their treatments, and it is noteworthy that failed replications of the watchful eye effect are relatively more common in studies which use stylized rather than realistic human eyes (Kelsey et al., 2018). It is conceivable that these features represent key differences across the sets of studies that render them incomparable or that account for the differences in the observed effects. Lastly, Matland and Murray (2019) and Murray (2019) put a lot of stock in the sign of a non-significant (negative) coefficient, which was opposite to ours in their experiments. Yet, given that their results are not significant and likely underpowered, this difference could largely (if not entirely) be due to sampling variability so we remain unconvinced by this finding.

In short, at first glance, Matland and Murray (2019) key conclusion seems reasonable: ‘[A]dding our three new tests weakens the conclusion that Republicans are more sensitive to watching eyes, but the effect is not so powerful that it conclusively undermines the previous result’ (p. 6). However, across our own studies – combined – we find that the difference in coefficients is significant between Republicans and Democrats. We are grateful that Matland and Murray (2019) challenged us to analyze our data in more rigorous ways and engaged in a debate that created the opportunity for us to illustrate the robustness of our results using a variety of methods. Although we recognize that sampling variability could account for some of the differences observed across the sets of studies (Gerber & Green, 2012), the results we report above only reinforced our initial conclusions that partisanship moderates voter responsiveness to watchful eyes and that Republicans appear to be more susceptible to watchful eyes than Democrats in the context of voting in elections.

Nonetheless, we recognize that neither of our studies explicitly manipulated partisanship experimentally, so we remain cautious about causality viewing our findings primarily as descriptive but intriguing. We acknowledge that although Democrats are also susceptible to social norms (Gerber, Green, & Larimer, 2008), our findings are consistent with a vast literature which shows that conservatives are still more susceptible to conformity and social influence than liberals (for a review, see Jost et al., 2018). Finally, we encourage future research to adjudicate and shed more light on important psychological differences in the political lives of liberals and conservatives.

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes

1. If we consider comparisons using the placebo conditions instead, the studies are even less powerful relative to the larger samples that leverage the pure control conditions.