Processing emotions from faces and words measured by event-related brain potentials

ABSTRACT

Affective aspects of a stimulus can be processed rapidly and before cognitive attribution, acting much earlier for verbal stimuli than previously considered. Aimed for specific mechanisms, event-related brain potentials (ERPs), expressed in facial expressions or word meaning and evoked by six basic emotions – anger, disgust, fear, happy, sad, and surprise – relative to emotionally neutral stimuli were analysed in a sample of 116 participants. Brain responses in the occipital and left temporal regions elicited by the sadness in facial expressions or words were indistinguishable from responses evoked by neutral faces or words. Confirming previous findings, facial fear elicited an early and strong posterior negativity. Instead of expected parietal positivity, both the happy faces and words produced significantly more negative responses compared to neutral. Surprise in facial expressions and words elicited a strong early response in the left temporal cortex, which could be a signature of appraisal. The results of this study are consistent with the view that both types of affective stimuli, facial emotions and word meaning, set off rapid processing and responses occur very early in the processing stage.

KEYWORDS:

• Emotions
• affective processing
• event-related potentials (ERPs)
• facial expressions
• emotional words

In everyday lives, we organise meaningful information around emotion categories. It is the aggregation of information, along with resolving any conflicts, that makes us arrive at an emotion category that best coincides with the sensory input and priorly acquired experience. Disentangling complex cognitive-emotional behaviours and their conceptualisation is an ongoing debate (Pessoa, 2008) and it is clear that perceiving and contemplating rapidly presented emotional information involves largely involuntary visceral, motor, and brain processes, which causally affect how emotional information is processed (Niedenthal, 2007). These processes may not be easily accessible by questionnaire-based methods and require other techniques, including various forms of brain imaging. Considering the role of emotions in social interaction, it is expected that an ability to perceive emotions from a person's facial expressions accurately has vital importance (Lange et al., 2022) and is often considered to receive preferential processing when compared to emotional words (Beall & Herbert, 2008). This also explains why facial expressions, along with emotional words, are among the most popular tools for studying affective information processing.

1.1. Emotions in words and faces

Using two different modes of presenting emotions – words and facial expressions – is a productive technique for separating components of emotion processing (Bayer & Schacht, 2014; Frühholz et al., 2011; Herbert et al., 2013; Rellecke et al., 2011; Schacht & Sommer, 2009). While facial expressions can be directly read from images, the emotional content of words can only be accessed after language processing. Unlike spoken words, written components are acquired relatively late in life. Because understanding word meaning is a sophisticated process, analysis of emotional meaning must be postponed in time relative to reading emotion from facial expressions. Because differences in word semantics start to appear in ERP around 250 ms reaching maximum values 400 ms after stimulus onset (Kutas & Federmeier, 2011), they are clearly suspended when compared to the first difference in ERP in response to facial expressions.

1.2. Dimensions of emotions expressed in faces and words

Perceived similarities and differences among emotions can define the dimensions where emotions are apparent. According to an influential theory, human emotions can be represented in a two-dimensional space with pleasure-displeasure and degree-of-arousal coordinates (Russell, 1980). There is evidence that the two dimensions characterise both facial expressions (Russell & Bullock, 1985) and emotion-denoting words (Russell et al., 1989). However, this does not mean ERPs characterising recognition of emotions in faces or words have a similar structure. It has been repeatedly observed that ERPs for words have a positive bias consisting of a stronger response to words denoting positive emotions (Bayer & Schacht, 2014; Herbert et al., 2008; Kissler et al., 2009; Schacht & Sommer, 2009). In the case of pictures; however, negative stimuli seem to attract preferential processing, resulting in augmented emotion effects for negative as compared to neutral or positive stimuli (Carretié et al., 2001; Kreegipuu et al., 2013; Olofsson et al., 2008).

Appraisal theories of emotion maintain that emotions are also driven by the results of cognitive appraisals and that the feeling cannot be adequately represented by two dimensions (Fontaine et al., 2007; Scherer & Fontaine, 2019). In addition to valence and arousal, dimensions of power and novelty are needed (Gillioz et al., 2016). However, these additional dimensions may not be required if used stimuli have a reduced variability on one of these dimensions. In other words, a high-dimensional semantic space may have different configurations adapting to the needs of a current task (Cowen & Keltner, 2021). For instance, facial expressions may form a space in which specific emotions, more than valence or arousal, are relevant in organising emotional experience, expression, and neural processing. As a suggestion for a research strategy, it recommends choosing specific emotions as basic categories for the analysis, instead of opting for a rigid contrast between positive and negative emotional content.

1.3. Appraisal in affective processing

While affective neuroscience emphasises the automatic mechanisms that can be routinely activated prior to cognitive processes, it cannot be eliminated that automatic affective mechanisms that arise either after or during the performance of semantic analysis for word meaning also exist. It is generally agreed that emotions are not simply an initial psychophysiological response to a triggering event, but also involve a subsequent mental process known as appraisal (Ellsworth, 2013; Moors, 2010; Scherer, 1999; Scherer et al., 2001). Although appraisal is typically conceptualised as cognitive control over emotions, analysis of its' neural substrate reveals that many subcortical emotion-generative systems are involved (Ochsner & Gross, 2005; Öner, 2018). Although appraisal is mainly considered a deliberate cognitive operation, it has also been noticed that automaticity can be developed, making it a more mechanical early process (Ellsworth, 2013; Moors, 2010; Thiruchselvam et al., 2011).

Automatic processing of emotions from faces seems to differ from the comprehension of word meaning, which is performed exclusively by learned and symbolic operations. Recent neurolinguistic studies have challenged the notion that affective processing, especially semantics, cannot be applied to language. Numerous examples have witnessed that emotions interact with semantic processing, which could occur in different brain regions (Hinojosa et al., 2020; van Berkum, 2019). Emotions and triggers igniting them, such as facial expressions, existed long before language developed, but triggers of emotions can hook up to new ones via learning (van Berkum, 2019). With the help of emotional conditioning, for example, the non-natural signs studied by linguistics can also instigate automatic affective processes in the brain (van Berkum, 2019). These affective processes, van Berkum notices (p. 8), are not necessarily intensive. For example, affective evaluation is a very subtle process that is not easy to notice (Zajonc, 1980). Affective judgements may be fairly independent of, and precede in time, the sorts of cognitive operations with which language is usually analysed. Thus, it is possible that emotional meaning activates affective judgements, which have their signatures in brain potentials.

1.4. Regions of interest

Complex networks of regions work together in the brain to process sensory information and enable perception and recognition. Early processing of visual information occurs mainly in the occipital lobe, with areas in the occipito-temporal cortex initially responding to facial expression analysis (Haxby et al., 2000). Lindquist et al. (2012) meta-analysis points to the anterior and medial temporal lobe, amygdala, insula and anterior midcingulate cortex as some of the consistently activated areas across studies assessing the experience or perception of emotional categories. The processing of words includes sublexical, phonological and lexico-semantic processing, generally considered to take place in the mid-fusiform gyus, dorsal and inferior supramarginal gyrus and middle and anterior temporal regions (see Hinojosa et al., 2020 for an overview). Lindquist et al. (2012) also report the left and right anterior lobes and right superior temporal cortex as areas commonly activated.

For enhanced ERPs attempting to capture emotional tasks in face processing parietal, parietal-occipital, occipito-temporal and temporal areas have been used (see Citron, 2012 for a review). Similarly, emotional tasks involving enhanced ERP amplitudes in word processing have been recorded from central (Schacht & Sommer, 2009), centro-parietal (Bayer & Schacht, 2014; Schacht & Sommer, 2009), parietal-temporal (Bayer & Schacht, 2014), occipito-temporal (Kissler et al., 2007, 2009) and occipito-parietal (Bayer & Schacht, 2014; Ponz et al., 2014) areas. While the occipital, parietal, and temporal areas are commonly activated by both visual and verbal stimuli, studies on emotional word processing typically incorporate central regions more frequently.

1.5. Research questions

The current study aimed to investigate three questions of interest. Firstly, we analysed differences in ERPs between stimuli representing one of the six basic emotions (anger, happiness, sadness, surprise, fear, disgust) and neutral stimuli as an indicator of specific emotion processing. Secondly, we presented participants with either pictures depicting facial expressions of six emotions or printed words conveying the emotion label (such as ‘anger,’ ‘happiness’, ‘sadness,’ etc.). Because previous studies have demonstrated a negative bias for facial expressions and a positive bias for emotion words, one of the aims was to test these biases. Thirdly, to differentiate between temporal variations in information processing in brain regions, we recorded ERPs from three regions of interest: the occipital, central, and left temporal areas. Given that emotion words require linguistic analysis, we anticipated that relatively delayed ERPs recorded from the left temporal area, linked with language understanding and symbolic information processing, would be involved. Likewise, based on previous research on recording sites, we expected the discrimination between emotional and neutral content for verbal stimuli to be more prominent in the central region. To our knowledge, we are the first to compare the electrocortical processing of emotional stimuli for the six basic emotions for both facial expressions and words.

2. Methods

2.1. Participants

119 healthy volunteers participated in the study. The data of 3 participants were excluded due to technical errors, and the final sample consisted of 116 participants (45 men; M = 25.02; SD = 6.34; age range: 18–49 years) with normal or corrected-to-normal eyesight. Sample size (105) requirements were calculated a priori with G*Power (Faul et al., 2009) ANOVA (repeated measures, within factors) model, using α error probability (0.05), power (1 -β error probability; 0.85), effect size (f ; 0.15), number of groups (7), number of measurements (2), correlation among repeated measures (0.5) and nonsphericity correction (ϵ; 1). The sample consisted of 91 Estonian, 17 Russian, and 8 English-speaking participants. The experiment was carried out in the primary language of Estonian and Russian-speaking participants. English-speaking participants had completed the TOEFL iBT, IELTS Academic, Cambridge English or PTE Academic test. 107 participants described themselves as right-handed, 8 left-handed, and 1 (main hand right) ambidextrous. A preliminary ANOVA on the self-reported task confidence ratings of the groups based on sex, primary language, and handedness showed no significant differences, and all groups were included in the final analyses. Participation was reimbursed with a 15 EUR department store gift card. The study was approved by the Research Ethics Committee of the University of Tartu per the ethical standards of the Code of Ethics of the World Medical Association (Declaration of Helsinki), and a written signed consent form was obtained from all participants.

2.2. Measures and procedure

The recordings were conducted in an electrically shielded quiet room in the Laboratory of Experimental Psychology at the University of Tartu. Stimuli were presented in Psychtoolbox (MATLAB, MathWorks) on a standard PC monitor (LCD, 19' diagonal). Throughout the experiment, subjective reports, measures of EEG, skin conductance (SCR), and video recordings of participants' facial expressions were recorded. Due to the scope of the current paper, aspects of the subjective reports, video material, and SCR will not be analysed further. Figure 1 illustrates the time course of a single trial in both conditions.

Figure 1. The general set-up of the face (A) and word (B) condition describing the time interval of a single trial in milliseconds (ms). The study consisted of two blocks: in block one, 42 depictions of an emotional or neutral face were shown, and in block two, 42 emotional or neutral words were presented for 6 ms. The picture depicting a face is an illustrative copyright-free stock photo and was not used in the experiment.

The facial expression stimuli were taken from the JACFEE dataset, a widely validated set of basic emotions including images of Japanese and Caucasian individuals (Matsumoto & Ekman, 1988) expressing six basic emotions: anger, disgust, happiness, sadness, surprise, and fear, and neutral facial expressions. The word stimuli included the words ‘anger’, ‘disgust’, ‘fear’, ‘happiness’, ‘sadness’, ‘surprise’, and ‘neutral’. The face condition block consisted of 7 (6 types of emotion and neutral) $\times 6$ (different variants in one emotion category) stimuli. Different variants of pictures in an emotion category were used to attain variability in emotional expressions, avoid habituation and minimise the impact of low-level visual features such as the presence of teeth. The word condition block consisted of 7 (6 types of emotion and neutral) $\times 6$ repetitions. There were 42 trials for each condition. All emotion stimulus responses were compared to a neutral stimulus, which was a neutral face in the facial expression condition and the word ‘neutral’ in the word condition.

Stimuli were semi-randomised (one set of 7 stimuli was shown in random order). In the first block, the participants viewed a target face picture for 6 s, mirrored the emotion displayed, and reported the difficulty of the task on a scale of 0–100. In the second block, the participants viewed a target word picture for 6 s, expressed the emotion, and reported the difficulty of the task on a scale of 0–100. The face condition was always presented before the word condition to avoid participants assigning a categorical label to the facial expression when first exposed to a categorical label (the word in the word condition). Self-reported ratings were used to evaluate task difficulty. A black and white grid was shown in-between pictures for 500 ms to reduce the sequential effects from rapid image playback.

Conventionally, words bearing emotional connotations are used when measuring the emotional processing of semantic concepts. Direct opposition of facial expressions of basic emotions with words with strong emotional associations, such as ”kiss” or ”kill,” may not be an adequate method of comparison, however. It has been shown that emotion-label and emotion-laden words elicit different cortical responses at both early (N170) and late (LPC) stages (Zhang et al., 2017). Traditionally, differentiating between basic emotions in studies is done by choosing appropriate labels. We aimed not to group an emotion face directly with an emotion word (thus, participants did not have to label emotional faces); however, to avoid differentiating levels of valence and arousal between the face and word condition, we opted to use emotion-label words (i.e. ‘happiness’, ‘sadness’, etc) for a more direct comparison between the JACFEE faces and verbal stimuli.

2.3. EEG recordings and pre-processing

EEG signals were recorded using a BioSemi ActiveTwo active electrode system consisting of 64 scalp electrodes, two reference electrodes behind the earlobes, and four ocular electrodes (above and below the left eye and near the outer canthi of both eyes). The EEG signals were recorded with a band-pass filter of 0.16–100 Hz, a sampling rate of 512 Hz, and following the international 10–20 system (Homan et al., 1987). The EEG data were analysed offline using BrainVision Analyzer 2.1 (Brain Products GmbH). The data was referenced to the earlobe electrodes, corrected for eye blinks (Gratton et al., 1983), and filtered using a Butterworth Zero Phase filter (0.16–30 Hz; 24 dB/oct). Data segments from $-200\mathrm{m}\mathrm{s}$ to $1000\mathrm{m}\mathrm{s}$ were extracted, and baseline correction made at 0–50 $\mathrm{m}\mathrm{s}$ post-stimulus onset. The average EEG activity in each interval for each electrode for each stimulus for each participant was calculated, and data points at 0 μV and higher than 100 μV were removed, resulting in a loss of 2.15% of raw data. ERPs were calculated by averaging the segments from the six stimulus repetitions from all participants $(N=116)$, resulting in 696 trials per one stimulus category per condition. ERPs were average-mastoid referenced.

A singular value decomposition principal component analysis (PCA) based on a correlation matrix was conducted to identify individual clusters of electrocortical activity. Three distinct clusters of cortical activity were observed -- one in the occipital and occipital-parietal region (referred to as the occipital region), one in the central and central-parietal area (referred to as the central region) and one in the left temporal region. No hemispatial lateralisation was observed in the occipital-parietal and central electrode areas. The first cluster that emerged as an area of interest was concentrated around the visual cortex (represented by electrodes P7, P5, P3, P1, PO7, PO3, POz, Oz, O1, P2, P4, P6, P8, PO4, PO8, O2), the second cluster involved the central region electrodes (C5, C3, C1, Cz, C2, C4, C6, CP5, CP3, CP1, CPz, Pz, CP2, CP4 and CP6), and the third cluster concerned the language processing area in the left temporal region (electrodes F7, FT7, T7, TP7 and P9).

2.4. Transparency and openness

We have reported determination for our sample size, all data exclusions, all manipulations, and measures in the study, and follow JARS guidelines (Kazak, 2018). All data, analysis code and research materials are available at https://osf.io/9yjgk. Data were analysed using R, version 4.2.2 (R Core Team, 2022) and the packages tidyverse (Wickham et al., 2019), lmerTest (Kuznetsova et al., 2017), emmeans (Lenth, 2021), eegkit (Helwig, 2018), and ERP (Causeur et al., 2019). This study's design and analysis were not pre-registered.

3. Results

3.1. Behavioral data

Self-reported confidence ratings for task success were lower in the face condition than for the word condition, as reflected in a main effect of condition, $F_{(1,9117)}=466.25,p<.0001,f=0.02)$. Self-reported confidence was also influenced by emotion F_(1,9117) = 177.09, p < .0001, f = 0.06) and by the interaction of emotion and condition F_(1,9117) = 16.72, p < .0001, f = 0.006). A boxplot with median confidence ratings and interquartile range can be seen in Figure 2.

Figure 2. Median self-reported confidence ratings of task success (0–100) for the face and word condition. The lower and upper hinges correspond to the first and third quartiles (the 25th and 75th percentiles).

A post hoc test on emotion with a Bonferroni correction for multiple testing revealed that for subjectively reported confidence ratings expressing all emotions besides fear was significantly easier in the word condition. Expressing anger was reported higher in the word (M = 69.61; SD = 18.85) than in the face condition (M = 64.13; SD = 17.73), t₍₆₆₀₎ = 7.22, p < .0001; disgust was reported higher in the word (M = 68.71; SD = 19.54) than in the face condition (M = 61.39; SD = 19.64), t₍₆₆₀₎ = 9.17, p < .0001; happiness was reported higher in the word (M = 76.36; SD = 16.33) than in the face condition (M = 67.87; SD = 18.55), t₍₆₆₀₎ = 12.1, p < .0001; neutral was reported higher in the word (M = 82.06; SD = 15.65) than in the face condition (M = 69.93; SD = 18.87), t₍₆₆₀₎ = 17.74, p < .0001; sadness was reported higher in the word (M = 67.14; SD = 19.44) than in the face condition (M = 61.93; SD = 19.16), t₍₆₆₀₎ = 7.1, p < .0001; and surprise was reported higher in the face (M = 69.67; SD = 19.1) than in the word condition (M = 64.21; SD = 18.17), t₍₆₆₀₎ = 7.36, p < .0001. The only emotion with no significant differences in confidence ratings of expression between the face (M = 58.88; SD = 20.51) and word condition (M = 60.48; SD = 23.65) was fear, t₍₆₆₀₎ = 1.85, p = .064. The confidence rating of expressing fear was also the lowest in both conditions.

3.2. Event-related brain potentials

We used a linear mixed model with restricted maximum likelihood estimation, Satterthwaite's estimate degrees of freedom and participant as a random effect to determine the general effect of emotion on the amplitude of the event-related potentials. For all statistical tests, the level of significance was set to 0.05. There was a significant main effect of emotion on the amplitude of the event-related potentials $(F_{(6,9937721)}=744.2,p<.001,f=0.02)$. Figure 3 depicts the general temporal progress of the ERP waves.

Figure 3. Grand average ERPs averaged in two conditions (face and word condition) and three pooled regions of interest (occipital, occipital-parietal and parietal marked as occipital, central and central-parietal marked as central, and left temporal). Grand average waves averaged into ROIs from all participants (N = 116) are time-locked to stimulus onset with the face condition depicted on the left and the word condition on the right. ERPs are average-mastoid referenced.

Additional point-wise significance analyses were conducted to investigate the initial moment of ERPs discriminating events carrying emotional content from events that are neutral towards emotional meaning. Data depicting the time intervals of interest (0–600 ms) of individual average ERP amplitudes were divided into 2 ms intervals and a time-varying coefficient multivariate regression model with fixed-time covariates implemented to test for the point-wise significance of an effect in a linear model. The model, including terms for the effect to be tested (emotion as a main effect and participant as random effect), was compared to a null model obtained by parameters specifying the effect set to zero (participant as random effect). The risk of erroneous identification of a significant time point, considering multiple comparisons and the high temporal dependence of residuals in linear model testing EEG data, was controlled by the Benjamini–Hochberg (BH) procedure (Benjamini & Hochberg, 1995). The false discovery rate (FDR) was controlled at a preset level (α = 0.05). Pairs of emotion and neutral were composed, the neutral condition subtracted from each emotion, and difference waves calculated to observe average differences of emotional content from the neutral stimuli (Figure 4). 301 pairwise F-tests were run on each emotion-minus neutral value in the occipital, central and left temporal ROIs. Due to the number of tests run, all pairwise F-test statistics, contrast effect confidence intervals, $R^2$ values, and FDR- and FWER-controlling multiple testing corrected p-values using the Benjamini–Hochberg (BH) procedure can be found in the below-listed OSF repository.

Figure 4. Emotion-minus-neutral ERP difference curves with significant F-test results (5% significance level) controlled for multiple comparisons and temporal dependence for faces and words in the (A) occipital, (B) central, and (C) left temporal regions.

Figure 4 demonstrates differences between ERPs to emotion-minus neutral face or word in the occipital, central and left temporal regions. The significant differences in the F-tests (5% significance level) are shown at the bottom of each graph for both faces and words. Figure 5 depicts the topographical distribution of emotion-minus-neutral ERPs at three time periods of interest.

Figure 5. Topographic maps of the grand average ERP difference signal emotion-minus-neutral faces and words for three subsequent time periods.

3.3. Results by emotion categories

Disgust and sad. ERPs elicited by disgust emerged steadily from 148–345 ms in the occipital and 343–600 ms in the central region, but not in the left temporal region in response to the face condition. For the disgust word, most significant responses appeared in the left temporal region. ERPs elicited by the sad face emerged inconsistently and for very short durations in the occipital, central, and left temporal region (for example, in the timeframe 73–103 ms in the occipital or 198–234 ms in the central region). ERPs elicited by the word ‘sad’ elicited a positive deflection in 527–600 ms in the occipital, and from 282–600 ms in the central region. No significant difference in the left temporal region was observed.

Anger. An angry face elicited an ERP in the occipital region 168–230 ms and 318–600 ms. In the central region, the significant differences were fragmented in the beginning and produced more stable negative differences in 305–600 ms. The word ‘angry’ produced somewhat consistent differences from the word ‘neutral’ from 65 ms onward in both the occipital and central region; however, only a fragmented difference was observed in the left temporal region (92–112 ms).

Fear. From about 70 ms after the onset of a fearful face, a very strong negative deflection of the ERP registered from the occipital and central region began to develop. In the left temporal region, a stable positive response emerged from 323 ms and continued until the end of the segment. On the contrary, the visual ERP in response to the word ‘fear’ was positive when compared to the ERP to the word ‘neutral’, and produced fragmented differences at 60–96 ms, 160–196 ms, 262–370 ms, and from 530 ms until the end of the segment. Similarly, the word ‘fear’ elicited two large positive ERP differences in the occipital region at 58–180 ms and 266–600 ms. In the left temporal region, there were two small significantly more positive segments at 94–122 ms and 192–264 ms for the word ‘fear’ than for ‘neutral’.

Happy. Both the happy face and the word ‘happiness’ produced a strong negative ERP compared to the neutral face and word meaning in the occipital and central regions. No significant differences for the ‘happy’ face or word ‘happy’ compared to neutral were found in the left temporal region.

Surprise. Surprise in the face elicited an ERP in the occipital region, which was significantly more negative than the ERP in response to a neutral face, with consistent differences emerging at 380 ms. Response to the word ‘surprise’ was fragmented. The same can be observed in the central region, where the face condition produced a significantly negative ERP most consistent at 146–600 ms. The word ‘surprise’ appeared fragmented with significant positivity emerging at 66–142 ms and 487–600 ms. A surprising find was made in the left temporal region where the surprised face and the word ‘surprise’ produce a significantly stronger positive ERP response when compared to the neutral face or word. The differences in the face condition emerged around 156–600 ms and for the word condition 60–600 ms.

4. Discussion

The present study aimed to examine differences in ERP presentations of emotions and neutral stimuli, compare emotional processing for facial expressions and words, and differentiate between processing areas by analysing data from the occipital, central, and left temporal areas. Emotion processing for facial expressions and words was successfully differentiated from neutral and processing revealed by ERPs was different for the specific emotions expressed either in facial expressions or word meaning, indicating both rapid and language-mediated processing. Self-reported confidence ratings indicated that all emotions, except fear, were more easily expressed in the word than in the face condition. Using a categorical label for emotion expression was significantly easier than simply mimicking a specified facial expression, a finding common in the previous literature (Palermo et al., 2013), although the result was not replicated for the elicitation of fear.

4.1. Affective processing

Affective processing means that emotion stimuli can induce congruent affective states, awareness of which could serve as an additional indicator of the emotional value of the presented stimuli. It is well-documented that pleasant or unpleasant pictures automatically induce specific affective reactions such as changes in heart rate and skin conductance (Codispoti et al., 2001). This study, together with numerous previous ones (Batty & Taylor, 2003; Bayer & Schacht, 2014; Frühholz et al., 2011; Herbert et al., 2013; Kreegipuu et al., 2013; Rellecke et al., 2011; Schacht & Sommer, 2009; Schupp et al., 2004; Uusberg et al., 2013), demonstrates that brain potentials can characterise automatic or relatively autonomous affective processes specific to a specific emotion category. We found remarkably different response patterns, which distinguish from one another starting from very early phases of processing and continuing to relatively late stages involving language processing. Our results are therefore in congruence with findings that emotional connotation can enhance cortical responses at all stages of emotional visual information processing.

4.2. Disgust and sad

Although disgust has been recognised as a basic emotion, it must also be a product of culture and learning because it is apparently absent in young children (Widen & Russell, 2013). Our study found that ERPs can effectively distinguish disgust from neutral very early in the processing stage. The differentiation of disgust from neutral was evident in the occipital and central regions between 148–345 ms and was particularly pronounced in the central area between 343–600 ms, a finding in line with previous studies (Batty & Taylor, 2003). Expressions of disgust are often mistaken for those of anger (Pochedly et al., 2012), but both the use of variability in facial expression stimuli and differences in self-reports and ERP responses indicate that the discrimination of these emotions from neutral was successful. The results regarding the emotion-label word ‘disgust’ showed only sparse differences from neutral in the occipital and central regions and a clear difference in the left temporal region at 320–538 ms. Our results could indicate a more semantic processing usually associated with the N400 component but this does not explain why no stable differences in the occipital and central regions were observed. A potential explanation is that the disgust emotion-label word isn't as salient a stimulus as disgust-laden words.

Previous studies have observed ERP components differentiating sad faces from neutral at 170 ms (Batty & Taylor, 2003), but no stable differences were found in our study. The bias for negative stimuli is well-documented in clinical patients where EEG studies have shown a greater ERP response for sad targets compared to neutral (Bistricky et al., 2014), specifically for groups with risk factors for depression. It is possible that the parts of neural circuitry observable by electrodes for processing sadness in non-clinical populations are not very different from those used for the analysis of faces in general without specified emotional expression. When facial expressions of different styles are projected into semantic space, the closest expression to neutral is sad (Hahn et al., 2019). The sad face may thus not be very well distinguishable from neutral unless there is a reason to be particularly alert to it, as observed in clinical populations.

4.3. Fear and happy: posterior negativity and parietal positivity

One of the key findings in emotion studies is the presence of a strong and early negative ERP in responses to negative facial emotions, particularly fear (Bayer & Schacht, 2014; Carretié et al., 2001; Kreegipuu et al., 2013; Olofsson et al., 2008; Rellecke et al., 2011; Schupp et al., 2004), as well as a later positive response to words denoting positive emotions (Bayer & Schacht, 2014; Herbert et al., 2008; Kissler et al., 2009; Schacht & Sommer, 2009). Results of this study by and large support these findings and provide additional details. The ERPs elicited by facial fear produced the largest deflection from zero, starting already around 70 ms and a strong positivity for the word ‘fear’ in the occipital and central regions. It has generally been proposed that processing pictorial material requires much greater attentional effort than processing words (Tempel et al., 2013). Alternatively, brain circuits responsible for detecting fear in faces could be involving the visual areas more intensively than the brain regions which participate in language processing. ERPs recorded from the occipital and central cortices detected differences between the verbal ‘fear’ and ‘neutral’ and just like the substantial ERP deviations observed in response to facial fear, the word ‘fear’ also created a departure of ERP from neutral. This negativity may be a brain signature of an appraisal process. It is not only fear in the face which triggers an early warning system, but also a word's emotional meaning that can cause an automatic process of affective evaluation. Although appraisal is typically thought of as a cognitive and deliberate process, automaticity can be developed in appraisal, and some spontaneous processes can be recruited through emotional conditioning (Ellsworth, 2013; Moors, 2010).

The word ‘happiness’ produced a significant ERP deflection already at about 80 ms post stimulus onset, coinciding with the time when a happy facial expression evoked a significant response. While semantic word processing is typically associated with the N400 ERP component (Kutas & Federmeier, 2011), previous studies have demonstrated early and late ERP sensitivity to emotional words (Herbert et al., 2008; Kissler et al., 2007). Some studies suggest differentiating emotion categories inferred from visual stimuli can act as fast or after than latencies required for object recognition (Kawasaki et al., 2001). Our results suggest that the emotional content of words, at least for some emotional categories, can be accessed as fast as facial expressions.

4.4. Anger and surprise: central effects and temporal positivity

Research suggests a preferential attentional and perceptual processing of threatening facial expressions (Kuldkepp et al., 2013; Rellecke et al., 2011; Schacht & Sommer, 2009; Schupp et al., 2004), reflecting them as particularly salient stimuli. We observed differences in response to angry faces in the occipital and central areas, consistent with previous research (Schacht & Sommer, 2009; Schupp et al., 2004). The rapid discrimination of angry faces from neutral supports the notion that the detection of angry expressions involves enhanced neural processing and automatic change detection, and thus such stimuli seem to be prioritised in early visual processing. Our study found that this preferential processing also applies to happy and fearful faces. The word ‘angry’ produced a similar neural response to the face stimulus in the occipital and central regions, indicating early processing. While semantic processing of words is typically observed around 400 ms post-stimulus onset, some studies have found that emotional connotations can be detected as early as 120 ms (Kissler et al., 2006). This suggests that a word's emotional connotation could be directly connected to the abstract representation of its visual form and emotional content amplifies early stages of semantic analysis in much the same way an instructed attention-enhancing processing task would.

Surprise falls between happiness and fear in the semantic space of emotions (Adolphs et al., 1996), it can be primed positively or negatively (Kim et al., 2004), and it is possible that positive and negative surprise may be two distinct emotions (Vrticka et al., 2014). The main components that differentiate surprise from all other emotions are novelty and unpredictability (Soriano et al., 2015). It is often accurately recognised in facial expression studies, and when mistaken, confused most frequently with fear, but rarely with happiness (Mill et al., 2009). Visual ERPs evoked by surprise in the face were similar to a strong negative deflection observed in response to fear, which seems to suggest that negative valence surprises are more expected than positive ones or surprise is initially coded as negative (Petro et al., 2018). Our study revealed a robust deviation in the processing of the word ‘surprise’ in the left temporal region from an early stage until the end of the segment. Soriano et al. (2015) argue that while all other basic emotions can be effectively described on a valence-arousal scale, surprise requires additional dimensions (novelty and power) to be incorporated into existing scales and the relationship between valence and arousal is regulated by the perceived uncertainty of valence (Brainerd, 2018). Studies on emotion-producing words often use a gross contrast of positive and negative words, but investigating specific emotional categories is often neglected. It is possible that the perceived ambiguity in the surprise word acted as a factor, requiring additional processing of the stimulus. The positive deflection in response to the word ‘surprise’ could be indicative of affective reappraisal, the primary function of which may be increasing alertness to unexpected events.

4.5. Limitations

Due to the nature of the experimental set-up, we were not able to collect direct measurements of perceived emotional stimuli. It should be taken into consideration that while the category label was perceived as easier than copying a facial expression, the self-reported confidence rating for task success may not offer the most direct means of comparing emotion faces and words. Consequently, the revealed differences in the self-report could be linked to the differences in mimicking an expression or translating a word to an emotion and then attempting to generate the associated expression. Thus, these differences may not solely stem from the differences between perceiving emotion faces and words.

5. Conclusions

The processing of different emotions was found to activate dissimilar mechanisms that provided an emotional processing of the situation. Self-reported confidence ratings of copying a facial expression or responding to a categorical emotional label were not associated with early ERP processing. Using a category label was almost always perceived easier (with the exception of fear), but this did not translate into ERP responses. Brain responses to sad faces or words in the occipital and left temporal regions did not differ from those to neutral stimuli. As previously documented, the presentation of a fearful face elicited early and strong posterior negativity. Contrary to expectations, both the happy face and word elicited significantly more negative responses compared to neutral stimuli in the form of parietal negativity. Both forms of surprise elicited a strong positive response in the left temporal brain region, which could reflect an appraisal process. The distinctive outcomes of the surprise stimuli indicate that in the study of emotional stimulus processing, it could be useful to utilise a broader range of emotional categories instead of relying solely on a valence-based division. The processing of happy, fearful, and angry emotional stimuli appeared early and stable in both processing faces and words, which indicates that emotional content is processed rapidly in both mediums. The results of this study are consistent with the view that both types of affective stimuli, facial emotions and word meanings, are able to set off rapid processing of emotional stimuli, acting as early for words as they do for faces. This indicates that both mediums have a similar potential to carry emotional meaning in communication.

Acknowledgments

We wish to thank Svetlana Bolotova, Dariia Temirova, Jelena Gorbova and Ikechukwu Ofodile for assisting in data collection, Egils Avots for help with the set-up of the experimental program and Stênio Foerster for his valuable input. We also thank the reviewers for their thorough comments and insights.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data availability statement

The data that support the findings of this study are openly available at OSF at https://doi.org/10.17605/OSF.IO/9YJGK, reference number 9YJGK.

Correction Statement

This article has been corrected with minor changes. These changes do not impact the academic content of the article.

Additional information

Funding

This work was funded by the Estonian Research Council grant IUT2-13 'Psychological mechanisms of cognition and personality processes’, PRG1151 ‘Pre-attentive information processing in the brain: relationship to state and trait variables, and behaviour’, and PUT638 'Enhancing human-computer interaction using multimodal emotion recognition based on gesture, face, and speech'.