Perceptual and affective mechanisms in facial expression recognition: An integrative review

Abstract

Facial expressions of emotion involve a physical component of morphological changes in a face and an affective component conveying information about the expresser’s internal feelings. It remains unresolved how much recognition and discrimination of expressions rely on the perception of morphological patterns or the processing of affective content. This review of research on the role of visual and emotional factors in expression recognition reached three major conclusions. First, behavioral, neurophysiological, and computational measures indicate that basic expressions are reliably recognized and discriminated from one another, albeit the effect may be inflated by the use of prototypical expression stimuli and forced-choice responses. Second, affective content along the dimensions of valence and arousal is extracted early from facial expressions, although this coarse affective representation contributes minimally to categorical recognition of specific expressions. Third, the physical configuration and visual saliency of facial features contribute significantly to expression recognition, with “emotionless” computational models being able to reproduce some of the basic phenomena demonstrated in human observers. We conclude that facial expression recognition, as it has been investigated in conventional laboratory tasks, depends to a greater extent on perceptual than affective information and mechanisms.

Keywords:

• Facial expression
• Recognition
• Emotion
• Affective priming
• Perception

Disclosure Statement

No potential conflict of interest was reported by the authors.

Notes

¹ The term expression “recognition” fits well within the categorical view, as it implies that there exists a discrete, “correct” emotion category with which a currently seen expression can be matched. In contrast, from the perspective of the dimensional-constructivist view, this process could be labelled expression “interpretation”, as there would be no fixed facial configurations for expressions, and emotion categories would be fuzzy. Rather than “accurate” recognition (categorical view), the process could thus be conceptualised as degree of “agreement” on an interpretation (dimensional-constructivist view). Nevertheless, for the sake of keeping a widely shared term, such as expression recognition, we will continue to use it, although keeping in mind the possible interpretive nature of the process.

² We conducted a statistical analysis (ANOVA followed by Bonferroni-corrected multiple post hoc comparisons) of the average matching scores, i.e., recognition agreement or “accuracy” across observers, reported in the Nelson and Russell (2013) review, with each set of data treated as a single case. The 17 cross-cultural judgement studies published between the years 1992 and 2010 were chosen, which provided 38 sets of data of Western (20) and non-Western (18) literate samples of observers. Statistically significant differences as function of expression emerged: scores were higher for happy faces (89%) than for all the other basic expressions, followed by surprise (83%), which were higher than for sadness and anger (71% and 68%, respectively), followed by disgust and fear (65% and 59%, respectively, which were similar to each other).

Additional information

Funding

This work was supported by the Spanish Ministerio de Ciencia e Innovación [grant number PSI2009-07245], and Ministerio de Economía y Competitividad [grant number PSI2014-54720-P] to MGC, and Academy of Finland [grant number 265917], and the European Research Council Starting [grant number 313000] to LN.