461
Views
0
CrossRef citations to date
0
Altmetric
Research Article

Emojis vs. facial expressions: An electrical neuroimaging study on perceptual recognition

& ORCID Icon
Pages 46-64 | Received 01 Nov 2022, Published online: 17 Apr 2023

References

  • Aguado, L., Martínez-García, N., Solís-Olce, A., Dieguez-Risco, T., & Hinojosa, J. A. (2018). Effects of affective and emotional congruency on facial expression processing under different task demands. Acta Psychologica, 187, 66–76. https://doi.org/10.1016/j.actpsy.2018.04.013
  • Akechi, H., Kikuchi, Y., Tojo, Y., Osanai, H., & Hasegawa, T. (2014). Neural and behavioural responses to face-likeness of objects in adolescents with autism spectrum disorder. Scientific Reports, 27(4), 3874. https://doi.org/10.1038/srep03874
  • Aluja, A., Balada, F., Blanco, E., Lucas, I., & Blanch, A. (2018). Startle reflex modulation by affective face “emoji” pictographs. Psychological Research: An International Journal of Perception, Attention, Memory, and Action, 84(1), 15–22. https://doi.org/10.1007/s00426-018-0991-x
  • Bai, Q., Dan, Q., Mu, Z., & Yang, M. (2019). A systematic review of emoji: Current research and future perspectives. Frontiers in Psychology, 10, 2221. https://doi.org/10.3389/fpsyg.2019.02221
  • Batty, M., & Taylor, M. J. (2003). Early processing of the six basic facial emotional expressions. Cognitive Brain Research, 17(3), 613–620. https://doi.org/10.1016/S0926-6410(03)00174-5
  • Beelen, C., Blockmans, L., Wouters, J., Ghesquière, P., & Vandermosten, M. (2022). Brain-behavior dynamics between the left fusiform and reading. Brain Structure & Function, 227(2), 587–597. https://doi.org/10.1007/s00429-021-02372-y
  • Bentin, S., Allison, T., Puce, A., Perez, E., & McCarthy, G. (1996). Electrophysiological studies of face perception in humans. Journal of Cognitive Neuroscience, 8(6), 551–565. https://doi.org/10.1162/jocn.1996.8.6.551
  • Bramão, I., Faísca, L., Forkstam, C., Reis, A., & Petersson, K. M. (2010). Cortical brain regions associated with color processing: An fMRI study. The Open Neuroimaging Journal, 4, 164–173. https://doi.org/10.2174/1874440001004010164
  • Chatzichristos, C., Morante, M., Andreadis, N., Kofidis, E., Kopsinis, Y., Theodoridis, S., & Davis, T. (2020). Emojis influence autobiographical memory retrieval from reading words: An fMRI-based study. Plos One, 15(7), 0234104. https://doi.org/10.1371/journal.pone.0234104
  • Cherbonnier, A., & Michinov, N. (2021). The recognition of emotions beyond facial expressions: Comparing emoticons specifically designed to convey basic emotions with other modes of expression. Computers in Human Behaviour, 118, 106689. https://doi.org/10.1016/j.chb.2021.106689
  • Churches, O., Nicholls, M. T., Keage, M., Kohler, H., & Keage, H. (2014). Emoticons in mind: An event-related potential study. Social Neuroscience, 9(2), 196–202. https://doi.org/10.1080/17470919.2013.873737
  • Cohen, L., Lehéricy, S., Chochon, F., Lemer, C., Rivaud, S., & Dehaene, S. (2002). Language-specific tuning of visual cortex? Functional properties of the Visual Word Form Area. Brain: A Journal of Neurology, 125(5), 1054–1069. https://doi.org/10.1093/brain/awf094
  • Comesaña, M., Soares, A. P., Perea, M., Piñeiro, A. P., Fraga, I., & Pinheiro, A. (2013). ERP correlates of masked affective priming with emoticons. Computers in Human Behavior, 29(3), 588–595. https://doi.org/10.1016/j.chb.2012.10.020
  • Dalle Nogare, L., Cerri, A., & Proverbio, A. M. (2023). Are men better at recognizing emojis than facial expressions? Preprints, 2023010403. https://doi.org/10.20944/preprints202301.0403.v1)
  • Daltrozzo, J., & Schön, D. (2009). Conceptual processing in music as revealed by N400 effects on words and musical targets. Journal of Cognitive Neuroscience, 21(10), 1882–1892. https://doi.org/10.1162/jocn.2009.21113
  • Davidson, R. J., Mednick, D., Moss, E., Saron, C., & Schaffer, C. E. (1987). Ratings of emotion in faces are influenced by the visual field to which stimuli are presented. Brain and Cognition, 6(4), 403–411. https://doi.org/10.1016/0278-2626(87)90136-9
  • Diéguez-Risco, T., Aguado, L., Albert, J., & Hinojosa, J. A. (2015). Judging emotional congruency: Explicit attention to situational context modulates processing of facial expressions of emotion. Biological Psychology, 112, 27–38. https://doi.org/10.1016/j.biopsycho.2015.09.012
  • Dozolme, D., Brunet-Gouet, E., Passerieux, C., Amorim, M. A., & Filik, R. (2015). Neuroelectric correlates of pragmatic emotional incongruence processing: Empathy matters. Plos One, 10(6), Article e0129770. https://doi.org/10.1371/journal.pone.0129770.
  • Duval, E. R., Moser, J. S., Huppert, J. D., & Simons, R. F. (2013). What’s in a face? The late positive potential reflects the level of facial affect expression. Journal of Psychophysiology, 27(1), 27. https://doi.org/10.1027/0269-8803/a000083
  • Eimer, M., & Holmes, A. (2007). Event-related brain potential correlates of emotional face processing. Neuropsychologia, 45(1), 15–31. https://doi.org/10.1016/j.neuropsychologia.2006.04.022
  • Ekman, P., & Friesen, W. V. (1971). Constants across cultures in the face and emotion. Journal of Personality and Social Psychology, 17(2), 124–129. https://doi.org/10.1037/h0030377
  • Fernández, A., Quiñónez, I., Bobes León, M. A., & Lage, A. (2008). 245. Affective valence of familiar faces modulates the P300 component. Clinical Neurophysiology, 119(9), 159. https://doi.org/10.1016/j.clinph.2008.04.261
  • Fischer, B., & Herbert, C. (2021). Emoji as affective symbols: Affective judgments of emoji, emoticons, and human faces varying in emotional content. Frontiers in Psychology, 20(12), 645173. https://doi.org/10.3389/fpsyg.2021.645173
  • Fonken, Y. M., Kam, J. W. Y., & Knight, R. T. (2020). A differential role for human hippocampus in novelty and contextual processing: Implications for p300. Psychophysiology, 57(7). https://doi.org/10.1111/psyp.13400
  • Gainotti, G. (2012). Unconscious processing of emotions and the right hemisphere. Neuropsychologia, 50(2), 205–218. https://doi.org/10.1016/j.neuropsychologia.2011.12.005
  • Gantiva, C., Sotaquirá, M., Araujo, A., & Cuervo, P. (2020). Cortical processing of human and emoji faces: An ERP analysis. Behaviour & Information Technology, 39(8), 935–943. https://doi.org/10.1080/0144929X.2019.1632933
  • Gosling, A., & Eimer, M. (2011). An event-related brain potential study of explicit face recognition. Neuropsychologia, 49(9), 2736–2745. https://doi.org/10.1016/j.neuropsychologia.2011.05.025
  • Greening, S. G., Mitchell, D. G. V., & Smith, F. W. (2018). Spatially generalizable representations of facial expressions: Decoding across partial face samples. Cortex, 101, 31–43. https://doi.org/10.1016/j.cortex.2017.11.016
  • Hadland, K. A., Rushworth, M. F., Gaffan, D., & Passingham, R. E. (2003). The effect of cingulate lesions on social behaviour and emotion. Neuropsychologia, 41(8), 919–931. https://doi.org/10.1016/s0028-3932(02)00325-1
  • Hagen, S., & Tanaka, J. W. (2019). Examining the neural correlates of within-category discrimination in face and non-face expert recognition. Neuropsychologia, 124, 44–54. https://doi.org/10.1016/j.neuropsychologia.2019.01.005
  • Hahn, A. C., Jantzen, K. J., & Symons, L. A. (2012). Thatcherization impacts the processing of own-race faces more so than other-race faces: An ERP study. Social Neuroscience, 7(2), 113–125. https://doi.org/10.1080/17470919.2011.583080
  • Halgren, E., Raij, T., Marinkovic, K., Jousmaki, V., & Hari, R. (2000). Cognitive response profile of the human fusiform face area as determined by MEG. Cerebral Cortex, 10(1), 69–81. https://doi.org/10.1093/cercor/10.1.69
  • Haxby, J. V., & Gobbini, M. I. (2011). Distributed neural systems for face perception. In A. Calder, G. Rhodes, M. Johnson, & J. Haxby (Eds.), Oxford handbook of face perception (pp. 93–110). Oxford University Press.
  • Hinojosa, J. A., Mercado, F., & Carretié, L. (2015). N170 sensitivity to facial expression: A meta-analysis. Neuroscience & Biobehavioral Review, 55, 498–509. https://doi.org/10.1016/j.neubiorev.2015.06.002
  • Hornak, J., Rolls, E. T., & Wade, D. (1996). Face and voice expression identification in patients with emotional and behavioural changes following ventral frontal lobe damage. Neuropsychologia, 34(4), 247–261. https://doi.org/10.1016/0028-3932(95)00106-9
  • Huang, A. H., Yen, D. C., & Zhang, X. (2008). Exploring the potential effects of emoticons. Information and Management, 45(7), 466–473. https://doi.org/10.1016/j.im.2008.07.001
  • Itier, R. J., Alain, C., Sedore, K., & McIntosh, A. R. (2007). Early face processing specificity: It’s in the eyes! Journal of Cognitive Neuroscience, 19(11), 1815–1826. https://doi.org/10.1162/jocn.2007.19.11.1815
  • Itier, R. J., & Taylor, M. J. (2002). Inversion and contrast polarity reversal affect both encoding and recognition processes of unfamiliar faces: A repetition study using ERPs. Neuroimage, 15(2), 353–372. https://doi.org/10.1006/nimg.2001.0982
  • Iwase, M., Ouchi, Y., Okada, H., Yokoyama, C., Nobezawa, S., Yoshikawa, E., Tsukada, H., Takeda, M., Yamashita, K., Takeda, M., Yamaguti, K., Kuratsune, H., Shimizu, A., & Watanabe, Y. (2002). Neural substrates of human facial expression of pleasant emotion induced by comic films: A PET Study. Neuroimage, 17(2), 758–768. PMID: 12377151.
  • Kadosh, K. C., Walsh, V., & Kadosh, R. C. (2011). Investigating face-property specific processing in the right OFA. Social Cognitive and Affective Neuroscience, 6(1), 58–65. https://doi.org/10.1093/scan/nsq015
  • Kanwisher, N., McDermott, J., & Chun, M. M. (1997). The Fusiform Face Area: A module in human extrastriate cortex specialized for face perception. Resource Policy, 17(11), 4302–4311. https://doi.org/10.1523/JNEUROSCI.17-11-04302.1997
  • Kestenbaum, R., & Nelson, C. A. (1992). Neural and behavioral correlates of emotion recognition in children and adults. Journal of Experimental Child Psychology, 54(1), 1–18. https://doi.org/10.1016/0022-0965(92)90014-W
  • Kilpatrick, L., & Cahill, L. (2003). Amygdala modulation of parahippocampal and frontal regions during emotionally influenced memory storage. Neuroimage, 20(4), 2091–2099. https://doi.org/10.1016/j.neuroimage.2003.08.006
  • Kim, K. W., Lee, S. W., Choi, J., Kim, T. M., & Jeong, B. (2016). Neural correlates of text-based emoticons: A preliminary fMRI study. Brain and Behavior, 6(8), 00500. https://doi.org/10.1002/brb3.500
  • Kirita, T., & Endo, M. (1995). Happy face advantage in recognizing facial expressions. Acta Psychologica, 89(2), 149–163. https://doi.org/10.1016/0001-6918(94)00021-8
  • Kume, Y., Maekawa, T., Urakawa, T., Hironaga, N., Ogata, K., Shigyo, M., & Tobimatsu, S. (2016). Neuromagnetic evidence that the right fusiform face area is essential for human face awareness: An intermittent binocular rivalry study. Neuroscience Research, 109, 54–62. https://doi.org/10.1016/j.neures.2016.02.004
  • Kutas, M., & Hillyard, S. A. (1980). Reading senseless sentences: Brain potentials reflect semantic incongruity. Science, 207(4427), 203–205. https://doi.org/10.1126/science.7350657
  • Lang, S. F., Nelson, C. A., & Collins, P. F. (1990). Event-related potentials to emotional and neutral stimuli. Journal of Clinical and Experimental Neuropsychology, 12(6), 946–958. https://doi.org/10.1080/01688639008401033
  • Liao, W., Zhang, Y., Huang, X., Xu, X., & Peng, X. (2021). “Emoji, I can feel your pain” - Neural responses to facial and emoji expressions of pain. Biological Psychology, 163. https://doi.org/10.1016/j.biopsycho.2021.108134
  • Liu, J., Harris, A., & Kanwisher, N. (2010). Perception of face parts and face configurations: An fMRI study. Journal of Cognitive Neuroscience, 22(1), 203–211. https://doi.org/10.1162/jocn.2009.21203
  • Liu, J., Kanwisher, N., Higuchi, M., & Marantz, A. (2000). The selectivity of the occipitotemporal M170 for faces. NeuroReport, 11(2), 337–341. https://doi.org/10.1097/00001756-200002070-00023
  • Lo, S. (2008). The nonverbal communication functions of emoticons in computer-mediated communication. CyberPsychology & Behavior, 11(5), 595–597. https://doi.org/10.1089/cpb.2007.0132
  • McPartland, J., Cheung, C. H., Perszyk, D., & Mayes, L. C. (2010). Face-related ERPs are modulated by point of gaze. Neuropsychologia, 48(12), 3657–3660. https://doi.org/10.1016/j.neuropsychologia.2010.07.020
  • Nigam, A., Hoffman, J. E., & Simons, R. F. (1992). N400 to semantically anomalous pictures and words. Journal of Cognitive Neuroscience, 4(1), 15–22. https://doi.org/10.1162/jocn.1992.4.1.15
  • Oleszkiewicz, A., Frackowiak, T., Sorokowska, A., & Sorokowski, P. (2017). Children can accurately recognize facial emotions from emoticons. Computers in Human Behavior, 76, 372–377. https://doi.org/10.1016/j.chb.2017.07.040
  • Oostenveld, R., & Praamstra, P. (2001). The five percent electrode system for high-resolution EEG and ERP measurements. Clinical Neurophysiology, 112(4), 713–719. https://doi.org/10.1016/s1388-2457(00)00527-7
  • Palmero-Soler, E., Dolan, K., Hadamschek, V., & Tass, P. A. (2007). swLORETA: A novel approach to robust source localization and synchronization tomography. Physics in Medicine and Biology, 52(7), 1783. https://doi.org/10.1088/0031-9155/52/7/002
  • Pascual-Marqui, R. D., Michel, C. M., & Lehmann, D. (1994). Low-resolution electromagnetic tomography: A new method for localizing electrical activity in the brain. International Journal of Psychophysiology, 18(1), 49–65. https://doi.org/10.1016/0167-8760(84)90014-x
  • Pitcher, D., Walsh, V., & Duchaine, B. (2011). The role of the occipital face area in the cortical face perception network. Experimental Brain Research, 209(4), 481–493. https://doi.org/10.1007/s00221-011-2579-1
  • Polich, J. (2020). 50+ years of p300: Where are we now? Psychophysiology, 57(7). https://doi.org/10.1111/psyp.13616
  • Poole, B. D., & Gable, P. A. (2014). Affective motivational direction drives asymmetric frontal hemisphere activation. Experimental Brain Research, 232(7), 2121–2130. https://doi.org/10.1007/s00221-014-3902-4
  • Prete, G., D’Ascenzo, S., Laeng, B., Fabri, M., Foschi, N., & Tommasi, L. (2015). Conscious and unconscious processing of facial expressions: Evidence from two split-brain patients. Journal of Neuropsychology, 9(1), 45–63. https://doi.org/10.1111/jnp.12034
  • Prete, G., Laeng, B., & Tommasi, L. (2014). Lateralized hybrid faces: Evidence of a valence-specific bias in the processing of implicit emotions. Laterality: Asymmetries of Body, Brain and Cognition, 19(4), 439–454. https://doi.org/10.1080/1357650X.2013.862255
  • Proverbio, A. M. (2021). Sexual dimorphism in hemispheric processing of faces in humans: A meta-analysis of 817 cases. Social Cognitive and Affective Neuroscience, 16(10), 1023–1035. https://doi.org/10.1093/scan/nsab043
  • Proverbio, A. M., Calbi, M., Manfredi, M., & Zani, A. (2015b). Audio-visuomotor processing in the Musician’s brain: An ERP study on professional violinists and clarinetists. Scientific Reports, 4(1), 5866. https://doi.org/10.1038/srep05866
  • Proverbio, A. M., Camporeale, E., & Brusa, A. (2020). Multimodal recognition of emotions in music and facial expressions. Frontiers in Human Neuroscience, 14(14), 32. https://doi.org/10.3389/fnhum.2020.00032
  • Proverbio, A. M., & Cerri, A. (2022). The recognition of facial expressions under surgical masks: the primacy of anger. Frontiers in Neuroscience, 16, 864490. https://doi.org/10.3389/fnins.2022.864490
  • Proverbio, A. M., Gabaro, V., Orlandi, A., & Zani, A. (2015a). Semantic brain areas are involved in gesture comprehension: An electrical neuroimaging study. Brain and Language, 147, 30–40. https://doi.org/10.1016/j.bandl.2015.05.002
  • Proverbio, A. M., & Galli, J. (2016). Women are better at seeing faces where there are none: An ERP study of face pareidolia. Social Cognitive and Affective Neuroscience, 11(9), 1501–1512. https://doi.org/10.1093/scan/nsw064
  • Proverbio, A. M., Tacchini, M., & Jiang, K. (2022, Nov 29). Event-related brain potential markers of visual and auditory perception: A useful tool for brain computer interface systems. Frontiers in Behavioral Neuroscience, 16, 1025870. https://doi.org/10.3389/fnbeh.2022.1025870
  • Richoz, A. R., Lao, J., Pascalis, O., & Caldara, R. (2018). Tracking the recognition of static and dynamic facial expressions of emotion across the life span. Journal of Vision, 18(9), 5. https://doi.org/10.1167/18.9.5
  • Rodrigues, D., Prada, M., Gaspar, R., Garrido, M. V., & Lopes, D. (2018). Lisbon Emoji and Emoticon Database (LEED): Norms for emoji and emoticons in seven evaluative dimensions. Behavior Research Methods, 50(1), 392–405. https://doi.org/10.3758/s13428-017-0878-6
  • Rolls, E. T., Critchley, H. D., Browning, A. S., & Inoue, K. (2006). Face-selective and auditory neurons in the primate orbitofrontal cortex. Experimental Brain Research, 170(1), 74–87. https://doi.org/10.1007/s00221-005-0191-y
  • Rossion, B. (2014). Understanding face perception by means of human electrophysiology. Trends in Cognitive Sciences, 18(6), 310–318. https://doi.org/10.1016/j.tics.2014.02.013
  • Rossion, B., & Gauthier, I. (2002). How does the brain process upright and inverted faces? Behavioral and Cognitive Neuroscience Reviews, 1(1), 63–75. https://doi.org/10.1177/1534582302001001004
  • Rossion, B., & Lochy, A. (2022). Is human face recognition lateralized to the right hemisphere due to neural competition with left-lateralized visual word recognition? A critical review. Brain Structure & Function, 227(2), 599–629. https://doi.org/10.1007/s00429-021-02370-0
  • Said, C. P., Moore, C. D., Engell, A. D., Todorov, A., & Haxby, J. V. (2010). Distributed representations of dynamic facial expressions in the superior temporal sulcus. Journal of Vision, 10(5), 11. https://doi.org/10.1167/10.5.11
  • Schindler, S., & Bublatzky, F. (2020). Attention and emotion: An integrative review of emotional face processing as a function of attention. Cortex, 30, 362–386. https://doi.org/10.1016/j.cortex.2020.06.010
  • Smailes, D., Burdis, E., Gregoriou, C., Fenton, B., & Dudley, R. (2020). Pareidolia-proneness, reality discrimination errors, and visual hallucination-like experiences in a non-clinical sample. Cognitive Neuropsychiatry, 25(2), 113–125. https://doi.org/10.1080/13546805.2019.1700789
  • Smith, A. P., Stephan, K. E., Rugg, M. D., & Dolan, R. J. (2006). Task and content modulate amygdala-hippocampal connectivity in emotional retrieval. Neuron, 49(4), 631–638. https://doi.org/10.1016/j.neuron.2005.12.025
  • Smith, E., Weinberg, A., Moran, T., & Hajcak, G. (2013). Electrocortical Responses to NIMSTIM Facial Expressions of Emotion. International Journal of Psychophysiology, 88(1), 17–25. https://doi.org/10.1016/j.ijpsycho.2012.12.004
  • Sun, A., Li, Y., Huang, Y. -M., Li, Q., & Lu, G. (2018). Facial expression recognition using optimized active regions. Human-Centric Computing and Information Sciences, 8(1), 1–24. https://doi.org/10.1186/s13673-018-0156-3
  • Tang, M., Chen, B., Zhao, X., & Zhao, L. (2020). Processing network emojis in Chinese sentence context: An ERP study. Neuroscience Letters, 722, 134815. https://doi.org/10.1016/j.neulet.2020.134815
  • Weiß, M., Mussel, P., & Hewig, J. (2020). The value of a real face: Differences between affective faces and emojis in neural processing and their social influence on decision-making. Social Neuroscience, 15(3), 255–268. https://doi.org/10.1080/17470919.2019.1675758
  • Willoughby, J. F., & Liu, S. (2018). Do pictures help tell the story? An experimental test of narrative and emojis in a health text message intervention. Computers in Human Behavior, 79, 75–82. https://doi.org/10.1016/j.chb.2017.10.031
  • Xiu, D., Geiger, M. J., & Klaver, P. (2015). Emotional face expression modulates occipital-frontal effective connectivity during memory formation in a bottom-up fashion. Frontiers in Behavioral Neuroscience, 23(9), 90. https://doi.org/10.3389/fnbeh.2015.00090
  • Yuasa, M., Saito, K., & Mukawa, N. (2006). Emoticons convey emotions without cognition of faces: An fMRI study. In Human factors in computing systems (CHI).
  • Yu, L., Xu, Q., Cao, F., Liu, J., Zheng, J., Yang, Y., & Zhang, L. (2022). Emotional violation of faces, emojis, and words: Evidence from N400. Biological Psychology, 173, 108405. https://doi.org/10.1016/j.biopsycho.2022.108405
  • Zhou, L., Jiang, C., Delogu, F., & Yang, Y. (2014). Spatial conceptual associations between music and pictures as revealed by N400 effect: Music conveys spatial concepts. Psychophysiology, 51(6), 520–528. https://doi.org/10.1111/psyp.12195

Reprints and Corporate Permissions

Please note: Selecting permissions does not provide access to the full text of the article, please see our help page How do I view content?

To request a reprint or corporate permissions for this article, please click on the relevant link below:

Academic Permissions

Please note: Selecting permissions does not provide access to the full text of the article, please see our help page How do I view content?

Obtain permissions instantly via Rightslink by clicking on the button below:

If you are unable to obtain permissions via Rightslink, please complete and submit this Permissions form. For more information, please visit our Permissions help page.