442
Views
15
CrossRef citations to date
0
Altmetric
Research Article

Automatic facial coding versus electromyography of mimicked, passive, and inhibited facial response to emotional faces

ORCID Icon, ORCID Icon, ORCID Icon & ORCID Icon
Pages 874-889 | Received 29 May 2020, Accepted 04 Mar 2021, Published online: 25 Mar 2021

References

  • Adolph, D., & Alpers, G. W. (2010). Valence and arousal: A comparison of two sets of emotional facial expressions. American Journal of Psychology, 123(2), 209–219. https://doi.org/10.5406/amerjpsyc.123.2.0209
  • Alpers, G. W., Adolph, D., & Pauli, P. (2011). Emotional scenes and facial expressions elicit different psychophysiological responses. International Journal of Psychophysiology, 80(3), 173–181. https://doi.org/10.1016/j.ijpsycho.2011.01.010
  • Alpers, G. W., & Gerdes, A. (2007). Here's looking at you: Emotional faces predominate in binocular rivalry. Emotion, 7(3), 495–506. https://doi.org/10.1037/1528-3542.7.3.495
  • Bartlett, M. S., Hager, J. C., Ekman, P., & Sejnowski, T. J. (1999). Measuring facial expressions by computer image analysis. Psychophysiology, 36(2), 253–263. https://doi.org/10.1017/S0048577299971664
  • Baur, R., Conzelmann, A., Wieser, M. J., & Pauli, P. (2015). Spontaneous emotion regulation: Differential effects on evoked brain potentials and facial muscle activity. International Journal of Psychophysiology, 96(1), 38–48. https://doi.org/10.1016/j.ijpsycho.2015.02.022
  • Berdica, E., Gerdes, A. B. M., Bublatzky, F., White, A. W., & Alpers, G. W. (2018). Threat vs. Threat: Attention to fear-related animals and Threatening faces. Frontiers in Psychology, 9, 1154. https://doi.org/10.3389/fpsyg.2018.01154
  • Beringer, M., Spohn, F., Hildebrandt, A., Wacker, J., & Recio, G. (2019). Reliability and validity of machine vision for the assessment of facial expressions. Cognitive Systems Research, 56, 119–132. https://doi.org/10.1016/j.cogsys.2019.03.009
  • Bijlstra, G., & Dotsch, R. (2011). FaceReader 4 emotion classification performance on images from the Radboud Faces Database. Unpublished manuscript, Department of Social and Cultural Psychology, Radboud University Nijmegen, Nijmegen, The Netherlands. http://www.gijsbijlstra.nl/ and http://ron.dotsch.org/.
  • Bishop, C. M. (2010). Neural networks for pattern recognition (Reprinted). Oxford Univ. Press.
  • Blairy, S., Herrera, P., & Hess, U. (1999). Mimicry and the judgment of emotional facial expressions. Journal of Nonverbal Behavior, 23(1), 5–41. https://doi.org/10.1023/A:1021370825283
  • Bonanno, G. A., Papa, A., Lalande, K., Westphal, M., & Coifman, K. (2004). The importance of being flexible: The ability to both enhance and suppress emotional expression predicts long-term adjustment. Psychological Science, 15(7), 482–487. https://doi.org/10.1111/j.0956-7976.2004.00705.x
  • Bradley, M. M., & Lang, P. J. (1994). Measuring emotion: The self-assessment manikin and the semantic differential. Journal of Behavior Therapy and Experimental Psychiatry, 25(1), 49–59. https://doi.org/10.1016/0005-7916(94)90063-9
  • Cacioppo, J. T., Berntson, G. G., Larsen, J. T., Poehlmann, K. M., & Ito, T. A. (2000). The psychophysiology of emotion. In M. Lewis, & J. M. Haviland-Jones (Eds.), Handbook of emotions (pp. 173–191). Guilford Press.
  • Calder, A. J., & Young, A. W. (2005). Understanding the recognition of facial identity and facial expression. Nature Reviews Neuroscience, 6(8), 641–651. https://doi.org/10.1038/nrn1724
  • Calvo, M. G., Fernández-Martín, A., Recio, G., & Lundqvist, D. (2018). Human observers and automated assessment of dynamic emotional facial expressions: KDEF-dyn database validation. Frontiers in Psychology, 9, 2052. https://doi.org/10.3389/fpsyg.2018.02052
  • Calvo, M. G., & Nummenmaa, L. (2016). Perceptual and affective mechanisms in facial expression recognition: An integrative review. Cognition and Emotion, 30(6), 1081–1106. https://doi.org/10.1080/02699931.2015.1049124
  • Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed). Lawrence Erlbaum Associates, Inc.
  • Cohn, J. F., & Sayette, M. A. (2010). Spontaneous facial expression in a small group can be automatically measured: An initial demonstration. Behavior Research Methods, 42(4), 1079–1086. https://doi.org/10.3758/BRM.42.4.1079
  • Cooney, R. E., Atlas, L. Y., Joormann, J., Eugène, F., & Gotlib, I. H. (2006). Amygdala activation in the processing of neutral faces in social anxiety disorder: Is neutral really neutral? Psychiatry Research: Neuroimaging, 148(1), 55–59. https://doi.org/10.1016/j.pscychresns.2006.05.003
  • Cordaro, D. T., Sun, R., Keltner, D., Kamble, S., Huddar, N., & McNeil, G. (2018). Universals and cultural variations in 22 emotional expressions across five cultures. Emotion, 18(1), 75–93. https://doi.org/10.1037/emo0000302
  • Dailey, M. N., Cottrell, G. W., Padgett, C., & Adolphs, R. (2002). EMPATH: A neural network that categorizes facial expressions. Journal of Cognitive Neuroscience, 14(8), 1158–1173. https://doi.org/10.1162/089892902760807177
  • Dimberg, U. (1982). Facial reactions to facial expressions. Psychophysiology, 19(6), 643–647. https://doi.org/10.1111/j.1469-8986.1982.tb02516.x
  • Dimberg, U., & Karlsson, B. (1997). Facial reactions to different emotionally relevant stimuli. Scandinavian Journal of Psychology, 38(4), 297–303. https://doi.org/10.1111/1467-9450.00039
  • Dimberg, U., Thunberg, M., & Elmehed, K. (2000). Unconscious facial reactions to emotional facial expressions. Psychological Science, 11(1), 86–89. https://doi.org/10.1111/1467-9280.00221
  • Dimberg, U., Thunberg, M., & Grunedal, S. (2002). Facial reactions to emotional stimuli: Automatically controlled emotional responses. Cognition & Emotion, 16(4), 449–471. https://doi.org/10.1080/02699930143000356
  • Dupré, D., Krumhuber, E. G., Küster, D., & McKeown, G. J. (2020). A performance comparison of eight commercially available automatic classifiers for facial affect recognition. Plos One, 15(4), e0231968. https://doi.org/10.1371/journal.pone.0231968
  • Durán, J. I., Reisenzein, R., & Fernández-Dols, J. M. (2017). Coherence between emotions and facial expressions. In J. M. Fernández-Dols, & J. A. Russel (Eds.), The Science of Facial expression (pp. 107–129). University Press. https://doi.org/10.1093/acprof:oso/9780190613501.001.0001.
  • Eisenbarth, H., Gerdes, A. B. M., & Alpers, G. W. (2011). Motor-incompatibility of facial reactions: The influence of valence and stimulus content on voluntary facial reactions. Journal of Psychophysiology, 25(3), 124–130. https://doi.org/10.1027/0269-8803/a000048
  • Ekman, P. (2017). Facial expression. In J. M. Fernández-Dols & J. A. Russel (Eds.), The Science of Facial expression (pp. 39–56). University Press. https://doi.org/10.1093/acprof:oso/9780190613501.001.0001.
  • Ekman, P., Friesen, W. V., & Hager, J. C. (2002). Facial action coding system. Manual and investigator’s guide. Research Nexus.
  • Faul, F., Erdfelder, E., Lang, A. G., & Buchner, A. (2007). G* Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39(2), 175–191. https://doi.org/10.3758/BF03193146
  • Fridlund, A. J., & Cacioppo, J. T. (1986). Guidelines for human electromyographic research. Psychophysiology, 23(5), 567–589. https://doi.org/10.1111/j.1469-8986.1986.tb00676.x
  • Greenaway, K. H., & Kalokerinos, E. K. (2019). The intersection of goals to experience and express emotion. Emotion Review, 11(1), 50–62. https://doi.org/10.1177/1754073918765665
  • Gross, J. J., & Levenson, R. W. (1993). Emotional suppression: Physiology, self-report, and expressive behavior. Journal of Personality and Social Psychology, 64(6), 970–986. https://doi.org/10.1037/0022-3514.64.6.970
  • Gross, J. J., & Levenson, R. W. (1997). Hiding feelings: The acute effects of inhibiting negative and positive emotion. Journal of Abnormal Psychology, 106(1), 95–103. https://doi.org/10.1037/0021-843X.106.1.95
  • Hess, U., & Blairy, S. (2001). Facial mimicry and emotional contagion to dynamic emotional facial expressions and their influence on decoding accuracy. International Journal of Psychophysiology, 19(2), 515–536. https://doi.org/10.1080/02699930441000364
  • Höfling, T. T. A., Gerdes, A., Föhl, U., & Alpers, G. W. (2020). Read my face: Automatic facial coding versus psychophysiological indicators of emotional valence and arousal. Frontiers in Psychology, 11, 1388. https://doi.org/10.3389/fpsyg.2020.01388
  • Huang, C. N., Chen, C. H., & Chung, H. Y. (2004). The review of applications and measurements in facial electromyography. Disability and Rehabilitation, 28(1), 231–237. https://doi.org/10.1080/09638280500158349
  • Iidaka, T., Matsumoto, A., Haneda, K., Okada, T., & Sadato, N. (2006). Hemodynamic and electrophysiological relationship involved in human face processing: Evidence from a combined fMRI–ERP study. Brain and Cognition, 60(2), 176–186. https://doi.org/10.1016/j.bandc.2005.11.004
  • Kappas, A., Bherer, F., & Thériault, M. (2000). Inhibiting facial expressions: Limitations to the voluntary control of facial expressions of emotion. Motivation and Emotion, 24(4), 259–270. https://doi.org/10.1023/A:1010718815960
  • Krohne, H., Egloff, B., Kohlmann, C., & Tausch, A. (1996). Untersuchungen mit einer deutschen Version der "Positive and Negative Affect Schedule"(PANAS) [Studies with a German version of the "Positive and Negative Affect Schedule"(PANAS)]. Diagnostica, 42(2), 139–156.
  • Krumhuber, E. G., Likowski, K. U., & Weyers, P. (2014). Facial mimicry of spontaneous and deliberate Duchenne and non-Duchenne smiles. Journal of Nonverbal Behavior, 38(1), 1–11. https://doi.org/10.1007/s10919-013-0167-8
  • Kulke, L., Feyerabend, D., & Schacht, A. (2020). A comparison of the affectiva iMotions facial expression analysis software with EMG for identifying facial expressions of emotion. Frontiers in Psychology, 11, 329. https://doi.org/10.3389/fpsyg.2020.00329
  • Küntzler, T., Höfling, T. T. A., & Alpers, G.W. (2021). Automatic facial expression recognition in standardized and non-standardized emotional expressions. Frontiers in Psychology. https://doi.org/10.3389/fpsyg.2021.627561
  • Lang, P. J., Greenwald, M. K., Bradley, M. M., & Hamm, A. O. (1993). Looking at pictures: Affective, facial, visceral, and behavioral reactions. Psychophysiology, 30(3), 261–273. https://doi.org/10.1111/j.1469-8986.1993.tb03352.x
  • Larsen, J. T., Norris, C. J., & Cacioppo, J. T. (2003). Effects of positive and negative affect on electromyographic activity over zygomaticus major and corrugator supercilii. Psychophysiology, 40(5), 776–785. https://doi.org/10.1111/1469-8986.00078
  • Laux, L., Glanzmann, P., Schaffner, P., & Spielberger, C. D. (1981). Das State-Trait-angstinventar [The State-Trait Anxiety inventory]. Beltz.
  • Lewinski, P. (2015). Automated facial coding software outperforms people in recognizing neutral faces as neutral from standardized datasets. Frontiers in Psychology, 6, 1386. https://doi.org/10.3389/fpsyg.2015.01386
  • Lewinski, P., den Uyl, T. M., & Butler, C. (2014). Automated facial coding: Validation of basic emotions and FACS AUs in FaceReader. Journal of Neuroscience, Psychology, and Economics, 7(4), 227–236. https://doi.org/10.1037/npe0000028
  • Likowski, K. U., Mühlberger, A., Gerdes, A., Wieser, M. J., Pauli, P., & Weyers, P. (2012). Facial mimicry and the mirror neuron system: Simultaneous acquisition of facial electromyography and functional magnetic resonance imaging. Frontiers in Human Neuroscience, 6, 214. https://doi.org/10.3389/fnhum.2012.00214
  • Lundqvist, D., Flykt, A., & Öhman, A. (1998). The Karolinska directed emotional faces (KDEF). CD ROM from Department of Clinical Neuroscience, Psychology Section, Karolinska Institute.
  • Matsumoto, D., & Hwang, H. C. (2018). Micro expressions differentiate truths from lies about future malicious intent. Frontiers in Psychology, 9, 2545. https://doi.org/10.3389/fpsyg.2018.02545
  • Mauss, I. B., & Robinson, M. D. (2009). Measures of emotion: A review. Cognition and Emotion, 23(2), 209–237. https://doi.org/10.1080/02699930802204677
  • Mavadati, S. M., Mahoor, M. H., Bartlett, K., Trinh, P., & Cohn, J. F. (2013). Disfa: A spontaneous facial action intensity database. IEEE Transactions on Affective Computing, 4(2), 151–160. https://doi.org/10.1109/T-AFFC.2013.4
  • Mohiyeddini, C., John, O., & Gross, J. J. (2008). Der "Berkeley Expressivity questionnaire" deutsche adaption und erste Validierungsbefunde [The berkley expressivity questionnaire, German adaption and first validation]. Diagnostica, 54(3), 117–128. https://doi.org/10.1026/0012-1924.54.3.117
  • Mourão-Miranda, J., Oliveira, L., Ladouceur, C. D., Marquand, A., Brammer, M., Birmaher, B., Axelson, D., Phillips, M. L., & Op de Beeck, H. P. (2012). Pattern recognition and functional neuroimaging help to discriminate healthy adolescents at risk for mood disorders from low risk adolescents. PloS One, 7(2), e29482. https://doi.org/10.1371/journal.pone.0029482
  • Neumann, R., Hess, M., Schulz, S., & Alpers, G. W. (2005). Automatic behavioural responses to valence: Evidence that facial action is facilitated by evaluative processing. Cognition and Emotion, 19(4), 499–513. https://doi.org/10.1080/02699930441000364.001
  • Neumann, R., Schulz, S. M., Lozo, L., & Alpers, G. W. (2014). Automatic facial responses to near-threshold presented facial displays of emotion: Imitation or evaluation? Biological Psychology, 96, 144–149. https://doi.org/10.1016/j.biopsycho.2013.12.009
  • Pantic, M., & Rothkrantz, L. J. M. (2000). Automatic analysis of facial expressions: The state of the art. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(12), 1424–1445. https://doi.org/10.1109/34.895976
  • Patterson, M. L. (2019). A systems model of dyadic nonverbal interaction. Journal of Nonverbal Behavior, 43(2), 111–132. https://doi.org/10.1007/s10919-018-00292-w
  • Pierce, C. A., Block, R. A., & Aguinis, H. (2004). Cautionary note on reporting eta-squared values from multifactor ANOVA designs. Educational and Psychological Measurement, 64(6), 916–924. https://doi.org/10.1177/0013164404264848
  • Plusquellec, P., & Denault, V. (2018). The 1000 most cited papers on visible nonverbal behavior: A bibliometric analysis. Journal of Nonverbal Behavior, 42(3), 347–377. https://doi.org/10.1007/s10919-018-0280-9
  • Poria, S., Cambria, E., Bajpai, R., & Hussain, A. (2017). A review of affective computing: From unimodal analysis to multimodal fusion. Information Fusion, 37, 98–125. https://doi.org/10.1016/j.inffus.2017.02.003
  • Reisenzein, R., Studtmann, M., & Horstmann, G. (2013). Coherence between emotion and facial expression: Evidence from laboratory experiments. Emotion Review, 5(1), 16–23. https://doi.org/10.1177/1754073912457228
  • Russell, J. A., & Bullock, M. (1985). Multidimensional scaling of emotional facial expressions: Similarity from preschoolers to adults. Journal of Personality and Social Psychology, 48(5), 1290–1298. https://doi.org/10.1037/0022-3514.48.5.1290
  • Rymarczyk, K., Żurawski, Ł, Jankowiak-Siuda, K., & Szatkowska, I. (2016). Do dynamic compared to static facial expressions of happiness and anger reveal enhanced facial mimicry? PLoS One, 11(7), e0158534. https://doi.org/10.1371/journal.pone.0158534
  • Rymarczyk, K., Żurawski, Ł, Jankowiak-Siuda, K., & Szatkowska, I. (2018). Neural correlates of facial mimicry: Simultaneous measurements of EMG and BOLD responses during perception of dynamic compared to static facial expressions. Frontiers in Psychology, 9, 52. https://doi.org/10.3389/fpsyg.2018.00052
  • Sato, W., Hyniewska, S., Minemoto, K., & Yoshikawa, S. (2019). Encoding of facial expressions of basic emotions in Japanese laypeople. Frontiers in Psychology, 10, 259. https://doi.org/10.3389/fpsyg.2019.00259
  • Scherer, K. R., & Ellgring, H. (2007). Multimodal expression of emotion: Affect programs or componential appraisal patterns? Emotion, 7(1), 158–171. https://doi.org/10.1037/1528-3542.7.1.158
  • Scherer, K. R., Ellgring, H., Dieckmann, A., Unfried, M., & Mortillaro, M. (2019). Dynamic facial expression of emotion and observer inference. Frontiers in Psychology, 10, 508. https://doi.org/10.3389/fpsyg.2019.00508
  • Scherer, K. R., & Moors, A. (2019). The emotion process: Event appraisal and component differentiation. Annual Review of Psychology, 70(1), 719–745. https://doi.org/10.1146/annurev-psych-122216-011854
  • Schneider, K. G., Hempel, R. J., & Lynch, T. R. (2013). That “poker face” just might lose you the game! The impact of expressive suppression and mimicry on sensitivity to facial expressions of emotion. Emotion, 13(5), 852–866. https://doi.org/10.1037/a0032847
  • Skiendziel, T., Rösch, A. G., & Schultheiss, O. C. (2019). Assessing the convergent validity between the automated emotion recognition software Noldus FaceReader 7 and Facial Action Coding System scoring. PloS One, 14(10), e0223905. https://doi.org/10.1371/journal.pone.0223905
  • Sonnby-Borgström, M., Jönsson, P., & Svensson, O. (2003). Emotional empathy as related to mimicry reactions at different levels of information processing. Journal of Nonverbal Behavior, 27(1), 3–23. https://doi.org/10.1023/A:1023608506243
  • Stangier, U., Heidenreich, T., Berardi, A., Golbs, U., & Hoyer, J. (1999). Die Erfassung sozialer Phobie durch die Social Interaction Anxiety Scale (SIAS) und die Social Phobia Scale (SPS) [Measurement of social phobia with the Social Interaction Anxiety Scale (SIAS) and the Social Phobia Scale (SPS)]. Zeitschrift für Klinische Psychologie und Psychotherapie, 28(1), 28–36. https://doi.org/10.1026//0084-5345.28.1.28
  • Stöckli, S., Schulte-Mecklenbeck, M., Borer, S., & Samson, A. C. (2018). Facial expression analysis with AFFDEX and FACET: A validation study. Behavior Research Methods, 50(4), 1446–1460. https://doi.org/10.3758/s13428-017-0996-1
  • Tassinary, L. G., Cacioppo, J. T., & Vanman, E. J. (2007). The skeletomotor system: Surface. In J. T. Cacioppo, L. G. Tassinary, & G. Berntson (Eds.), Handbook of psychophysiology (pp. 267–299). University Press.
  • Tian, Y. I., Kanade, T., & Cohn, J. F. (2001). Recognizing action units for facial expression analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 23(2), 97–115. https://doi.org/10.1109/34.908962
  • Tottenham, N., Tanaka, J. W., Leon, A. C., McCarry, T., Nurse, M., Hare, T. A., Marcus, D. J., Westerlund, A., Casey, B. J., & Nelson, C. (2009). The NimStim set of facial expressions: Judgments from untrained research participants. Psychiatry Research, 168(3), 242–249. https://doi.org/10.1016/j.psychres.2008.05.006
  • Van Kuilenburg, H., Den Uyl, M. J., Israël, M. L., & Ivan, P. (2008). Advances in face and gesture analysis. In Proceedings of 6th International Conference on Methods and Techniques in Behavioral Research (pp. 371–372).
  • Viola, P., & Jones, M. J. (2004). Robust real-time face detection. International Journal of Computer Vision, 57(2), 137–154. https://doi.org/10.1023/B:VISI.0000013087.49260.fb
  • Vrana, S. R. (1993). The psychophysiology of disgust: Differentiating negative emotional contexts with facial EMG. Psychophysiology, 30(3), 279–286. https://doi.org/10.1111/j.1469-8986.1993.tb03354.x
  • Vuilleumier, P., Armony, J. L., Driver, J., & Dolan, R. J. (2001). Effects of attention and emotion on face processing in the human brain: An event-related fMRI study. Neuron, 30(3), 829–841. https://doi.org/10.1016/S0896-6273(01)00328-2
  • Wild, B., Erb, M., & Bartels, M. (2001). Are emotions contagious? Evoked emotions while viewing emotionally expressive faces: Quality, quantity, time course and gender differences. Psychiatry Research, 102(2), 109–124. https://doi.org/10.1016/S0165-1781(01)00225-6
  • Wolf, K., Mass, R., Ingenbleek, T., Kiefer, F., Naber, D., & Wiedemann, K. (2005). The facial pattern of disgust, appetence, excited joy and relaxed joy: An improved facial EMG study. Scandinavian Journal of Psychology, 46(5), 403–409. https://doi.org/10.1111/j.1467-9450.2005.00471.x
  • Yitzhak, N., Giladi, N., Gurevich, T., Messinger, D. S., Prince, E. B., Martin, K., & Aviezer, H. (2017). Gently does it: Humans outperform a software classifier in recognizing subtle, nonstereotypical facial expressions. Emotion, 17(8), 1187–1198. https://doi.org/10.1037/emo0000287
  • Yoon, K. L., & Zinbarg, R. E. (2008). Interpreting neutral faces as threatening is a default mode for socially anxious individuals. Journal of Abnormal Psychology, 117(3), 680–685. https://doi.org/10.1037/0021-843X.117.3.680
  • Yoshikawa, S., & Sato, W. (2006). Enhanced perceptual, emotional, and motor processing in response to dynamic facial expressions of emotion. Japanese Psychological Research, 48(3), 213–222. https://doi.org/10.1111/j.1468-5884.2006.00321.x
  • Zhang, X., Yin, L., Cohn, J. F., Canavan, S., Reale, M., Horowitz, A., Liu, P., & Girard, J. M. (2014). Bp4d-spontaneous: A high-resolution spontaneous 3d dynamic facial expression database. Image and Vision Computing, 32(10), 692–706. https://doi.org/10.1016/j.imavis.2014.06.002
  • Zung, W. W. (1965). A self-rating depression scale. Archives of General Psychiatry, 12(1), 63–70. https://doi.org/10.1001/archpsyc.1965.01720310065008

Reprints and Corporate Permissions

Please note: Selecting permissions does not provide access to the full text of the article, please see our help page How do I view content?

To request a reprint or corporate permissions for this article, please click on the relevant link below:

Academic Permissions

Please note: Selecting permissions does not provide access to the full text of the article, please see our help page How do I view content?

Obtain permissions instantly via Rightslink by clicking on the button below:

If you are unable to obtain permissions via Rightslink, please complete and submit this Permissions form. For more information, please visit our Permissions help page.