Watching eyes and partisan mobilization: A rejoinder to Panagopoulos and van der Linden

ABSTRACT

This rejoinder addresses concerns raised by Panagopoulos and van der Linden about replication studies of their work conducted by Matland and Murray and published in this journal.Specifically, it offers counterarguments grounded in a broader view of the evidence to the assertion and findings that watching eyes stimuli more effectively mobilize Republican/conservative identifiers to vote than Democratic/liberal identifiers. It concludes that the overall evidence generated by the original and replication research is inconclusive at best.

KEYWORDS:

• Watching eyes
• implicit social cues
• voter turnout
• replication
• ideology

In a 2016 article, Panagopoulos and van der Linden presented a hypothesis with supporting empirical results that implicit social pressure in the form of watching eyes mobilizes Republican/conservative political identifiers at a higher rate than Democratic/liberal political identifiers. In a 2019 follow-up article, Matland and Murray reported conceptual replications of Panagopoulos and van der Linden that failed to reproduce Panagopoulos and van der Linden’s results either statistically or substantively. Panagopoulos and van der Linden (2019) responded by presenting additional analyses of their data and noting concerns about Matland and Murray’s empirical approach, in particular a lack of statistical power and underappreciated differences between the studies. This rejoinder addresses the concerns raised about Matland and Murray’s replications by arguing that Panagopoulos and van der Linden ignore the broader view of the results of these studies. Further, it concludes by reasserting that the evidence in support of an effect of watching eyes on partisan mobilization is inconclusive at best.¹

Opposite effect: When statistical power is immaterial

In their response, Panagopoulos and van der Linden (2019) argue that the Matland and Murray (2019) replication studies lack adequate statistical power to detect reliably the hypothesized effects. The results reported in Table 1, which summarizes Tables 2 and 3 from Matland and Murray (2019), suggest that statistical power is immaterial. The table presents the experimental treatment effects of watching eyes stimuli on voter turnout at their three reported experimental sites. At two of the three sites (Midland, Texas, and Toledo, Ohio), the direct test of the hypothesis reported in Column 4 shows the effect of watching eyes on Republicans compared to Democrats is substantively negative (−0.050 and −0.011). Put otherwise, the hypothesis suggests these differences should be positively signed but the results reported here are negatively signed. Further, Column 2 shows the effect at the two sites is also substantively negative using Panagopoulos and van der Linden’s relaxed test, which compares the effect on Republicans to the effect on the control group. So, with even unlimited statistical power, the hypothesized effect is not only unsupported by these results it is largely contradicted by them. Republicans may be demobilized by watching eyes compared to Democrats in these studies.

Table 1. Treatment effects on partisan turnout for implicit social pressure mailers at four sites.

TREATMENT & SITE	REP	DEM	DIFFERENCE: REP-DEM	CHOW TEST: DIFFERENCE
Watching Eyes
El Paso, TX
Treatment	.009 (.031)	.008 (.017)	.001 (.056)	Chi^2 = .01 p = .92
Placebo	.016 (.086)	.020 (.025)
N	6,192	20,671
RMSE	.500	.500
Midland, TX
Treatment	−.002 (.017)	.048 (.033)	−.050 (.034)	Chi^2 = 1.96 p = .16
Placebo	.014 (.023)	.030 (.040)
N	5,728	1,499
RMSE	.499	.461
Toledo, OH
Treatment	−.007 (.024)	.004 (.014)	−.011 (.026)	Chi^2 = .59 p = .44
Placebo	−.059* (.033)	−.033* (.020)
N	8,881	28,207
RMSE	.473	.497
Self-Mailer
Lubbock, TX
Treatment	.011 (.015)	.038 (.024)	−.027 (.027)	Chi^2 = .87 p = .35
Civic Duty	−.009 (.015)	.058* (.025)
N	19,099	8,790
RMSE	.447	.499

Estimates are unstandardized regression coefficients (standard errors in parentheses) derived using bivariate OLS regression. They represent average intent-to-treat effects (ITT). The dependent variable is voter turnout. Covariate-adjusted estimates only vary marginally. See Matland and Murray (2019) for more details.

* Statistically significant at p < .05, one-tailed test.

Matland and Murray also report the results of another social pressure mobilization field experiment conducted in Lubbock, Texas, that uses a different social pressure technique to test the hypothesis. If the broader theoretical assertion is correct that party identification moderates the effect of social pressure, then one would expect other social pressure stimuli also to spur greater turnout among Republicans than Democrats. In lieu of watching eyes, this experiment treated subjects with a so-called ‘self-mailer,’ which reports to each subject her or his voter turnout history. As Table 1 indicates, the effect of this different social pressure technique on Republicans directly compared to Democrats is also substantively negative (−0.027) even though the hypothesis indicates it should be positive. Again, with unlimited statistical power, the hypothesized effect is not only unsupported in the direct test of the hypothesis it is contradicted.

Overall, then, Panagopoulos and van der Linden’s concern about a lack of statistical power for the Matland and Murray replication studies is unfounded. From this broader view, null hypothesis significance testing is not the issue. Contrary to the hypothesis, in two of the three watching eyes studies the effect on Democrats was greater than it was on Republicans, and when including the self-mailer study, the effect on Democrats was greater than Republicans in three of the four studies. The results presented here suggest that unlimited statistical power would reveal effects that are largely opposite to the hypothesis and their results.

Between-study differences and non-generalizability

Panagopoulos and van der Linden also suggest that the argument presented by Matland and Murray undervalues the differences between the studies in regard to sampling, treatments, and methodologies. As with almost all replications in social science (Schmidt, 2009), there are differences between the original field experiments. But because many successful voter mobilization techniques identified in field experiments are later used in actual campaigns (e.g., Erkel, 2018; Issenberg, 2013), Matland and Murray intentionally designed their original studies (2016) as conceptual versus exact or direct replications (Schmidt, 2009). In particular, the project was designed as a constructive replication (Lykken, 1968), a type of conceptual replication, to test the effect of watching eyes (or, more broadly, social pressure) on voter turnout. It was not designed to test narrowly the effect of Panagopoulos’s specific eye images on his specific sample in his specific electoral context. In other words, Matland and Murray placed a high priority on generalizability. Both teams of authors identify and discuss several operational differences related to sampling and experimental procedures. Not surprisingly, Panagopoulos and van der Linden (2019) express greater concern over these differences than Matland and Murray (2019). Readers will have to decide for themselves whether the differences are consequential or not.

Regardless, arguments that between-study differences account for the lack of support for the proposed hypothesis are also arguments that Panagopoulos and van der Linden’s results are not generalizable. As Lykken notes, ‘ … a theory of any useful degree of predictive richness should in principle allow for sufficient empirical confirmation through multiple corroboration [i.e., the derivation and testing of a number of separate, quasi-independent predictions] to compel the respect of the most critical reader or editor’ (1968, pp. 154–155). Using different electoral contexts (e.g., voter turnout, office, the electorate’s partisanship, or political culture; see Matland & Murray, 2019, for details) and different experimental procedures (e.g., watching eye images, sampling, or partisanship identification), Matland and Murray are unable to verify the hypothesis and corroborate the result that watching eyes mobilize Republican voters at a relatively greater rate than Democratic voters. This suggests Panagopoulos and van der Linden’s findings, in the best case, are of limited generalizability.

Conclusion

Panagopoulos and van der Linden (2016) presented a hypothesis and supporting analyses showing results that Matland and Murray (2019) failed to replicate substantively or statistically. Panagopoulos and van der Linden (2019) responded to the failed replications with further analyses in support of their position. This rejoinder follows with evidence and counterarguments that invoke a broader and still skeptical view of the effect of watching eyes on partisan mobilization. Their arguments regarding statistical power and between-study differences remain open to interpretation. The broader view invoked here suggests that statistical power is immaterial in the replication studies and that the best case for Panagopoulos and van der Linden may be that they found results that are not generalizable. Overall, the evidence generated by these studies is inconclusive, and readers will have to decide what utility they assign to watching eyes as a tool for partisan mobilization. Hopefully, this exchange will motivate researchers to investigate partisan mobilization further.

Professors Panagopoulos and van der Linden should be applauded for initiating this research and then engaging in a thoughtful and informative exchange about it. Scholars and practitioners know more about partisan mobilization as a result of their efforts. Moreover and importantly, the editors of Social Influence should be commended for facilitating this exchange and dedicating scarce pages of their journal to it. Few journals openly embrace replication (Martin & Clarke, 2017; Yeung, 2017) despite its vital role in the advance of science (Open Science Collaboration, 2015).

Disclosure statement

No potential conflict of interest was reported by the author.

Additional information

Funding

This work was supported by the Donald P. Green.

Notes

1. Professor Matland passed away shortly after Matland and Murray (2019) was initially submitted to this journal. While this rejoinder was drafted in a manner believed to be consistent with his spirit and approach, Professor Matland’s insightful analyses and zealous yet collegial pursuit of the scientific endeavor were sorely missed in this effort and will be sorely missed for years to come.