A sad thumbs up: incongruent gestures and disrupted sensorimotor activity both slow processing of facial expressions

ABSTRACT

Recognising a facial expression is more difficult when the expresser's body conveys incongruent affect. Existing research has documented such interference for universally recognisable bodily expressions. However, it remains unknown whether learned, conventional gestures can interfere with facial expression processing. Study 1 participants (N = 62) viewed videos of people simultaneously producing facial expressions and hand gestures and reported the valence of either the face or hand. Responses were slower and less accurate when the face-hand pairing was incongruent compared to congruent. We hypothesised that hand gestures might exert an even stronger influence on facial expression processing when other routes to understanding the meaning of a facial expression, such as with sensorimotor simulation, are disrupted. Participants in Study 2 (N = 127) completed the same task, but the facial mobility of some participants was restricted, which disrupted face processing in prior work. The hand-face congruency effect from Study 1 was replicated. The facial mobility manipulation affected males only, and it did not moderate the congruency effect. The present work suggests the affective meaning of conventional gestures is processed automatically and can interfere with face perception, but does not suggest that perceivers rely more on gestures when sensorimotor face processing is disrupted.

KEYWORDS:

• Emotion perception
• gesture
• sensorimotor simulation

Acknowledgments

We thank the individuals who participated in this study, and we thank Magdalena Rychlowska, Nolan Lendved, Leah Schultz, Nathanael Smith, Crystal Hanson, Olivia Zhao, Emma Phillips, Holden Wegner, Alicia Waletzki, Mathias Hibbard, and Jay Graiziger for their help with stimuli creation and validation and data collection. We also thank John Lendved for filming and editing the video stimuli, and Mark Koranda for his useful insights on ASL.

Disclosure statement

No potential conflict of interest was reported by the authors.

ORCID

Adrienne Wood http://orcid.org/0000-0003-4773-4493

Notes

1. What constitutes a “congruent” face-hand pairing depends on the salient feature dimension; in the current study participants focused on the valence of the facial expressions and hand gestures. The facial expressions and gestures might also be re-categorized according to other dimensions, such as approach-avoidance, resulting in a different set of face-hand stimuli pairs being considered “congruent” vs. “incongruent.” We operationalised congruency by valence, rather than by specific emotions (e.g. angry faces and angry gestures), because it is unclear what conventional gestures, if any, would convey the same specific meanings as discrete facial expressions.

2. A coding error that was not discovered until both studies were completed resulted in all participants repeating a block of gesture-attend trials at the end of the session. Since this occurred at the end of the session, we simply excluded these redundant trials from our analyses. Including them in the analyses did not change any of our conclusions.

3. The means and standard deviations reported are averaged across participants and are therefore not sensitive to within-subject changes in reaction times. This is why the effect can be statistically significant despite the means being so close together.

Additional information

Funding

A.W. and J.D.M. were supported by NIH Emotion Research Training Grant (T32MH018931-24). Further support for this research was provided by the Office of the Vice Chancellor for Research and Graduate Education at the University of Wisconsin–Madison with funding from the Wisconsin Alumni Research Foundation; National Institute of Mental Health.