Facing the Electorate: Computational Approaches to the Study of Nonverbal Communication and Voter Impression Formation

ABSTRACT

Politicians have strong incentives to use their communication to positively impress and persuade voters. Yet one important question that persists within the fields of political science, communication, and psychology is whether appearance or substance matters more during political campaigns. To a large extent, this appearance vs. substance question remains open and, crucially, the notion that appearance can in fact effectively sway voter perceptions is consequential for the health of democracy. This study leverages advances from the fields of machine learning and computer vision to expand our knowledge on how nonverbal elements of political communication influence voters immediate impressions of political actors. We rely on video from the 4th Republican Party presidential debate held on November 10, 2016, as well as continuous response approval data from a live focus group (n = 311; 36,528 reactions), to determine how the emotional displays of political candidates influence voter impression formation. Our results suggest that anger displays can positively influence viewers’ real-time evaluations. Happiness displays, on the other hand, are much less effective in eliciting a response from the viewing public, while fear displays were rarely projected by the candidates of the debate under study.

KEYWORDS:

• Political communication
• nonverbal communication
• machine learning
• computer vision

Disclosure Statement

No potential conflict of interest was reported by the authors.

Supplementary Material

Supplemental data for this article can be accessed on the publisher’s website at https://doi.org/10.1080/10584609.2020.1784327.

Notes

1. Kissel, Rick (2015) “GOP Debate Ratings: Fox Business Network Draws Record 13.5 Million Viewers.” Variety, accessed at https://bit.ly/2KKYJMl

2. RealClearPolitics, “2016 Republican Presidential Nomination,” accessed at https://bit.ly/2iiYMSZ

3. Speaking time was as follows, in descending order: Ted Cruz (13.6 min), John Kasich (11.9 min), Donald Trump (11.3 min), Carly Fiorina (11 min), Marco Rubio (10.3 min), Rand Paul (10.1 min), Jeb Bush (9.8 min), and Ben Carson (9.4 min).

4. We uploaded 10 photos of each of the eight candidates (Donald Trump, Ben Carson, Marco Rubio, Ted Cruz, Jeb Bush, Carly Fiorina, John Kasich, and Rand Paul) and three journalists (Maria Bartiromo, Gerard Baker, and Neil Cavuto) to the Microsoft Identify API to train a person identification model. To validate the accuracy of the Identity API, the authors annotated a random sample of 250 frames, coding for the presence of the eight candidates and three moderators. If the frame included a face other than the candidates or moderators, the face was coded as “unknown,” while frames without a face were coded as “None”. Frames with multiple faces (e.g., more than two candidates) were labeled to include each face that was present. The results of this exercise suggest that the face detection model was extremely accurate: with a micro-averaged precision of 0.96, recall of 0.94, and an F1-score of 0.95. This model was ultimately used to identify each face in a given frame and provide a confidence score for the match.

5. Note that we also were able to recover the head position of the detected face in each frame (yaw, roll, and pitch); however, we do not use this information in the current study.

6. Debate participant segments were coded manually by a researcher using the following decision rules. A new segment begins when a candidate is first heard or seen. Likewise, the segment ends when the same candidate is finished answering a question and there are no follow-up questions from one of the journalists. If there is a follow-up question, this time is included in the segment. In the very rare circumstance that another candidate interrupts, we decided to attribute the segment to the original candidate unless the interruption is prolonged, whereby the segment changes to the new candidate.

7. In particular, the themes include Climate change, Closing statements, Entitlements, Financial crisis and bailouts, Hillary Clinton, Immigration, Income inequality, Media portrayal of candidates, Minimum wage, National debt and federal spending, National security and foreign affairs, Regulations on businesses, Tax loopholes and businesses leaving, Taxes, Technology impact on jobs, and the Trans-Pacific Partnership (TPP). These topics were derived from an initial list of topics generated by Ballotpedia (see https://bit.ly/2VHYvv4)

8. Political and demographic covariates are measured as follows: Republican (1 = Republican; 0 = Independents), age dummies (18–24, 25–29, 30–39, 40–49, 50–65, 65+), race dummies (African-American, Asian, Hispanic, Other, White), and female (1 = female, 0 = male).

9. In principle, the lag distribution may be estimated directly by incorporating a separate regressor for each second (1 to 4). In practice, however, multicollinearity makes direct estimation difficult (if not impossible) and thus it is common practice to place constrain the lag distribution to promote smoothness. Following past scholarship, we assume a quadratic lag function:

${\beta }_s={\lambda }_0+{\lambda }_{1}s+{\lambda }_2{s}^2,s=0,\cdots ,4,$

which we can then plug into the (ordered logistic) regression equation:

$y_t=\alpha +\sum _{s=0}^4({\lambda }_0+{\lambda }_{1}s+{\lambda }_2{s}^2)x_{t-s}+{ϵ}_t,$

which one can re-write as:

$y_t=\alpha +{\lambda }_0{z}_t^0+{\lambda }_1{z}_t^1+{\lambda }_2{z}_t^2+{ϵ}_t,$

where

$z_t^0=\sum _{s=0}^4{x}_{t-s},z_t^1=\sum _{s=0}^{4}s{x}_{t-s},z_t^2=\sum _{s=0}^4{s}^2{x}_{t-s}.$

We can then estimate (4) by standard models and recover the lag weights using (1). Uncertainty of the estimates can be assessed by examining the linear combination of coefficients expressed in (1).

Additional information

Funding

This work was supported by the Trinity College Dublin [Arts and Social Sciences Benefactions Fund 2018-19]. Travis G. Coan’s contribution was supported by the UK Economic and Social Research Council [ES/N012283/1].

Notes on contributors

Constantine Boussalis

Constantine Boussalis is Assistant Professor of Political Science at Trinity College Dublin. His research lies at the intersection of computational social science, political communication, and political behavior, with a particular emphasis on the communication of environmental politics and policy.

Travis G. Coan

Travis G. Coan is Senior Lecturer in Quantitative Politics at the University of Exeter, where he is co-director of the Q-Step Centre. His research focuses on environmental and political communication, specifically in the areas of climate change and violent extremism.