Emotion recognition of faces and emoji in individuals with moderate-severe traumatic brain injury

ABSTRACT

Background

Facial emotion recognition deficits are common after moderate-severe traumatic brain injury (TBI) and linked to poor social outcomes. We examine whether emotion recognition deficits extend to facial expressions depicted by emoji.

Methods

Fifty-one individuals with moderate-severe TBI (25 female) and fifty-one neurotypical peers (26 female) viewed photos of human faces and emoji. Participants selected the best-fitting label from a set of basic emotions (anger, disgust, fear, sadness, neutral, surprise, happy) or social emotions (embarrassed, remorseful, anxious, neutral, flirting, confident, proud).

Results

We analyzed the likelihood of correctly labeling an emotion by group (neurotypical, TBI), stimulus condition (basic faces, basic emoji, social emoji), sex (female, male), and their interactions. Participants with TBI did not significantly differ from neurotypical peers in overall emotion labeling accuracy. Both groups had poorer labeling accuracy for emoji compared to faces. Participants with TBI (but not neurotypical peers) had poorer accuracy for labeling social emotions depicted by emoji compared to basic emotions depicted by emoji. There were no effects of participant sex.

Discussion

Because emotion representation is more ambiguous in emoji than human faces, studying emoji use and perception in TBI is an important consideration for understanding functional communication and social participation after brain injury.

KEYWORDS:

• Traumatic brain injury
• emotion recognition
• facial affect recognition
• emoji
• faces

Introduction

Social cognition deficits are common after moderate-severe traumatic brain injury (TBI). They can include deficits in emotion recognition, theory of mind, cognitive empathy, and pragmatic inference (1) and have the potential to negatively affect social functioning in everyday communication settings. In particular, deficits in emotion recognition in people with TBI have been linked to poorer social integration outcomes (2) and reduced communication competence (3). Studies of emotion recognition in TBI have provided important insights into the difficulties people with TBI encounter in face-to-face communication. However, not all communication is face-to-face. Increasingly, people engage in social communication via written media such as text messages, e-mail, and posts on social media. People with TBI communicate using a variety of platforms including social networking sites, instant messaging, dating apps, gaming, and e-mail (4) and report that participation in social media serves important social functions such as connecting with others, sharing TBI stories, advocacy, and creating community (4–6). Although people with TBI use social media in ways that are similar to those of their uninjured peers, a subset of participants with TBI report a change in usage following their brain injury (7) as well as significant cognitive and technological barriers in their use of computer-mediated communication (CMC) more generally (4,8). Although CMC contexts lack in-person facial cues, written text is often accompanied by emoji – stylized graphical icons depicting facial expressions as well as other imageable ideas such as hand gestures, weather, food and drink, animals, plants, and activities. In the first investigation of emoji perception after brain injury, we examine whether the documented deficits in facial emotion recognition in TBI extend to recognition of emotions depicted by emoji, providing insights into the communicative competence of people with TBI and potential barriers to social participation in CMC contexts.

Emotion recognition of faces in TBI

Across cultures, people recognize at least six basic human emotions: anger, fear, sadness, disgust, happiness, and surprise (9–11). The universality of emotion perception has been established in cross cultural studies in which participants match facial expressions to a closed-set of emotion labels; however, cultural variation has been detected when using an open response format in which participants freely generated emotion words without cues (12). There is evidence that TBI can disrupt recognition of basic emotions; a meta-analysis of 13 studies found that between 13% and 39% of people with moderate-severe TBI have deficits in facial emotion recognition of static basic emotions, performing on average 1.1 standard deviations below neurotypical peers (13). Facial emotion recognition deficits have been linked to social communication difficulties in TBI (3,14) and a reduction in perceived communication competence by close others (15). However, people with TBI are not uniformly impaired at facial emotion recognition. Research into the risk factors associated with poor emotion recognition after brain injury suggest that injury severity (16–18); location of lesion to gray and/or white matter (19,20); neuropsychological profile (21,22); and age (22,23) may contribute to emotion recognition impairments. In addition, biological sex may play a protective role against emotion recognition deficits (24): females with TBI performed significantly better than males with TBI in a dynamic facial recognition task, and males (but not females) with TBI performed significantly worse than neurotypical peers in dynamic facial emotion recognition (22). Participant age and sex have also been linked to emotion recognition abilities in neurotypical people, with younger participants outperforming older participants and females outperforming males (25–29).

Emotion recognition of emoji

Although decades of research on facial emotion recognition have established the importance of facial expression for communication, research into the recognition of graphical representations of emotion such as emoji is in its early stages. In CMC contexts, emoji provide a nonverbal cue that parallels the role of facial expression in face-to-face communication. Although many emoji are devised to depict facial expressions, the extent to which emoji map onto specific emotions is unclear. Indeed, descriptions of a given emoji often span multiple emotions (30,31), suggesting that their interpretation is flexible. For example, Jaeger et al 31 categorized participants’ free-text emoji labels into 44 semantic categories and found that although some emoji had strong associations with a particular emotion (e.g., 99% of responses for the Grinning-face emoji were coded as “happy”), other emoji generated a variety of semantic category responses (e.g., 57% of responses for the Confused-face emoji were coded as “sad” whereas 5–28% of its responses were coded as categories such as “confused,” “disappointed,” “neutral,” “depressed,” or “nervous”). One additional challenge to interpreting emoji is the different representations of emoji across platforms. Although emoji have a standard Unicode identifier, each communication platform (e.g., iOS, Android, Facebook, Twitter) has its own style of emoji icons. As a result, a sender using one platform (e.g., iOS) may select a specific emoji that reflects their mood, but the receiver, using a different platform (e.g., Android), sees a different representation. Indeed, although potential for multiple interpretations exists within a platform, inter-platform differences in emoji depiction represent an even greater risk for miscommunication (32,33). Franco and Fugate (34) asked participants to select up to three emoji that most accurately represented a variety of emotions (anger, calm, contempt, disgust, envy, fear, happy, love, sad, and surprise) and to rate the strength of the emoji-emotion relationship. The researchers compared responses within and across three platforms (iOS, Android, and Samsung) and found limited shared agreement on which emoji best represented these emotions, with the exception of the Angry-face emoji that was widely agreed to depict the emotion anger.

Given the lack of clear emoji-emotion relationships, Cherbonnier and Michinov (35) created a novel set of emoji specifically designed to represent the six basic emotions previously identified by Ekman (10) (anger, sad, disgust, surprise, happy, fear). Across three studies, the researchers compared emotion recognition for these novel emoji with Facebook emoji, iOS emoji, and human faces. For each stimulus item, participants selected the best-fitting emotion label from a bank of 14 emotion words (the six basic emotions and eight filler emotions). The novel emoji had superior emotion recognition compared to Facebook emoji, iOS emoji, and human faces. The authors attributed the improved recognition accuracy of the novel emoji to the disgust emotion, which was depicted with a tongue sticking out and had significantly higher recognition rates and perceived intensity than the disgust emotions depicted by the Facebook and iOS emoji. Indeed, current emoji seem to lack a clear parallel for the facial expression disgust; using a corpus of 150 most popular emoji on Twitter (both face and non-face emoji), Shoeb and de Melo (36) asked participants to rate the degree of association of each emoji with eight emotions (anger, anticipation, disgust, fear, joy, sadness, surprise, and trust). They found that few emoji were strongly associated with the basic emotions, except for joy, which had many associations to both face and object emoji. In particular, disgust was not strongly associated with any emoji, although it was moderately associated with a few (e.g., Confounded-face ). The Confounded-face emoji was also found to be most frequently associated with disgust in Franco and Fugate (34). Examining all basic emotions besides disgust, Fischer and Herbert (37) compared the emotionality (i.e., how intensely a stimulus represents an emotion) of human faces and emoji and found no difference in emotionality between the two stimulus types, indicating that they represent associated emotions similarly. However, there were some differences for specific emotions. Fear and neutral were rated as having higher emotionality for faces than emoji, whereas anger and sadness had higher emotionality for emoji than faces. Although these studies provide an emerging picture of the relationship between emoji and emotion perception in neurotypical individuals, perception of emoji by people with TBI has not been examined.

Demographic variables that influence emoji perception

Demographic variables such as sex and age may influence emoji perception. Some studies report that females use emoji more frequently than males (38–40), but males use a larger variety of types (41). However, there is mixed evidence for whether sex differences exist in emoji perception. Jones and colleagues (40) found that compared to males, females rated emoji depicting negative emotions as having significantly lower valence, and in another study, females rated emoji as being more familiar, clear, and meaningful than males did (42). In contrast, males and females did not differ in their types or frequency of responses when selecting emotion labels associated with commonly-used emoji (43) or in their interpretation of emoji functions (39). Thus, although males and females may differ in their patterns of emoji use or familiarity, they may perceive semantic meanings associated with emoji similarly.

In a study of age differences in emoji use, older participants reported using emoji less frequently, had less positive attitudes toward emoji, and identified fewer motives for using them than younger participants (38). Regarding emoji perception, older participants were more likely than younger participants to respond “I don’t know” when asked to interpret emoji functions and interpreted emoji more literally (39). However, others (43) found no influence of age on emoji interpretation. These studies (38,39,43) treated age as a categorical variable, leaving an open question as to the age-related effect size when age is treated as a continuous variable. Given documented sex and age differences in emotion recognition in faces (22,23,25–28), continued work is needed to understand the demographic factors that affect emotion recognition in emoji.

Study aims

The current study had four aims:

1. Replicate findings of reduced facial emotion recognition accuracy of static basic human emotions in a novel comparably-sized sample of individuals with moderate-severe TBI.

In a sample of 53 people with moderate-severe TBI, Rigon and colleagues (22) found that participants with TBI had significantly reduced accuracy labeling basic emotions depicted by static human faces than their neurotypical peers. Further, female participants showed higher facial recognition accuracy of static faces depicting basic emotions than males. There was no significant sex-by-group interaction for static facial recognition tasks. Given the hallmark heterogeneity of TBI (44), we set out to test whether these findings hold in a novel comparably-sized and sex-balanced sample of people with moderate-severe TBI, recruited from a different testing site. Although not a direct replication due to stimulus presentation differences (in-person versus Zoom), we predicted that like the participants in Rigon et al, this new sample of participants with TBI would also demonstrate poorer facial emotion recognition than neurotypical peers and that males would demonstrate lower emotion label accuracy than females.

• (2) Compare emotion recognition accuracy of emotions depicted by emoji and human faces.

Given that some emoji can represent multiple emotions (30,31,34), we predicted that both participants with and without TBI would have reduced emotion labeling accuracy for emoji relative to human faces. We hypothesized that facial emotion recognition deficits in moderate-severe TBI would extend to the emoji modality, and thus, predicted a group-by-stimulus condition interaction where the TBI group would demonstrate a greater reduction in emotion labeling accuracy of emoji than human faces relative to the neurotypical group.

• (3) Compare emotion recognition accuracy for emoji depicting basic versus social emotions.

To date, studies examining emoji-emotion relationships have focused on the six basic emotions initially identified by Ekman and colleagues (34–37). There are over 100 facial expression emoji represented by unique Unicode identifiers and depicted across platforms (https://unicode.org/emoji/charts/full-emoji-list.html). Given that senders frequently use facial emoji depicting a variety of emotions and moods, we compared emotion recognition for emoji depicting the six basic emotions (anger, disgust, fear, sadness, surprise, happy) as well as six social emotions (embarrassed, remorseful, anxious, flirting, confident, proud). Social emotions differ from basic emotions in that they require a social context, involving interactions with other people and inferences about others’ mental states (45–49). For example, feelings of embarrassment require a social context where a person becomes self-conscious about another’s real or imagined negative perceptions. We predicted that both groups would have reduced emotion labeling accuracy for emoji depicting social emotions relative to basic emotions. However, given that people with TBI commonly have deficits in social cognition (1), we predicted a group-by-stimulus condition interaction where the TBI group would demonstrate a greater reduction in emotion labeling accuracy of social emotions relative to basic emotions than the neurotypical group.

• (4) Compare confidence ratings of emotion labels by stimulus type and group.

If emoji represent emotion more ambiguously than human faces, we would expect all participants to be less confident in matching emotion labels to emoji than human faces. Moderate-severe TBI can result in anosognosia (50), a lack of deficit awareness, and those with reduced self-awareness tend to perform lower on emotion recognition tests (51,52). Thus, although we predicted participants with TBI will have reduced emotion labeling accuracy relative to neurotypical peers, we did not predict a parallel decrease in emotion labeling confidence ratings in the TBI group.

Methods

Participants

Participants were 51 individuals with moderate-severe TBI (25 Female) and 51 neurotypical peers (26 Female). TBI and neurotypical groups were matched on sex (X² [1, N = 102] = 0.04, p = 0.84), age (M_TBI = 38.53 years, SD_TBI = 10.60 years; M_Neurotypical = 37.51 years, SD_Neurotypical = 10.73; t = −0.48, p = 0.63, d = 0.10), and education (M_TBI = 14.84 years, SD_TBI = 2.44 years; M_Neurotypical = 15.45 years, SD_Neurotpyical = 2.19; t = 1.32, p = 0.19, d = 0.26).

Participants with TBI sustained their injuries in adulthood, were in the chronic phase of their injury, at least 6 months since onset of injury (mean time post injury = 77.64 months; SD = 78.24 months), and were out of post-traumatic amnesia, exhibiting stable neuropsychological profiles. All participants with TBI met inclusion criteria for moderate-severe TBI, determined using the Mayo Classification System (53), along with health history information collected through medical records and intake interviews. Participants were classified as moderate-severe if at least one of the following criteria was met: (1) Glasgow Coma Scale (GCS) <13 within 24 hours of acute care admission (i.e., moderate or severe according to the GCS), (2) positive neuroimaging findings (acute CT findings or lesions visible on a chronic MRI), (3) loss of consciousness (LOC) >30 minutes, or (4) post-traumatic amnesia (PTA) >24 hours. GCS was available for 41 participants (Median = 12, ranging from 3 to 15); loss of consciousness (LOC) information was available for 40 participants; post-traumatic amnesia (PTA) information was available for 41 participants; acute imaging information was available for 36 participants (35 with positive findings). Causes of injury were motor vehicle accidents (n = 21), falls (n = 9), motorcycle or snowmobile accidents (n = 5), being hit by a car as a pedestrian (n = 3), non-motorized vehicle accidents (n = 6), assault (n = 2), being hit by a moving object (n = 2), or other (e.g., hang gliding accident, fell out of moving car; n = 3). See Table 1 for demographic and injury characteristics of the TBI sample.

Table 1. Demographic and injury information for participants with TBI.

ID	Age	Edu	Etiology	TSO	LOC	Neuroimaging	GCS	PTA
5002	41–45	16	Non-motorized vehicle accident	255	LOC >30 minutes	ICH	3	>24 hours
5003	26–30	18	Ped vs. auto	52	Unknown or N/A	SDH	11	>24 hours
5014	51–55	16	MVA	213	LOC >30 minutes	N/A	N/A	>24 hours
5016	18–25	16	MVA	44	LOC >30 minutes	SAH	13	>24 hours
5017	31–35	16	Ped vs. auto	198	LOC >30 minutes	SAH; IVH	4	>24 hours
5018	36–40	18	MVA	176	LOC >30 minutes	SAH	3	>24 hours
5021	41–45	18	MVA	60	LOC >30 minutes	Epidural hematoma; SAH	3	>24 hours
5027	31–35	16	Ground-level fall	40	LOC >30 minutes	SAH	9	>24 hours
5029	31–35	14	Non-motorized vehicle accident	40	LOC < 30 minutes	SDH; IPH; SAH	14	< 24 hours
5034	36–40	16	MVA	61	LOC >30 minutes	SAH	3	>24 hours
5038	41–45	16	Ground-level fall	46	LOC >30 minutes	SDH; multifocal hemorrhages; post-traumatic hemorrhagic contusions	N/A	>24 hours
5041	31–35	16	MVA	81	No LOC	Negative	10	>24 hours
5046	46–50	18	Non-motorized vehicle accident	73	LOC < 30 minutes	SAH	14	>24 hours
5047	26–30	16	Assault	42	LOC < 30 minutes	SDH	15	< 24 hours
5048	46–50	16	MVA	365	LOC >30 minutes	N/A	N/A	>24 hours
5050	31–35	18	Ground-level fall	42	LOC >30 minutes	SAH; IPH	15	< 24 hours
5051	51–55	16	MVA	26	LOC <30 minutes	SAH; SDH	14	<24 hours
5058	31–35	12	MCC	133	LOC < 30 minutes	SAH; SDH; PCH	8	>24 hours
5068	26–30	16	Fall from height	62	LOC < 30 minutes	SDH; epidural hematoma	3	>24 hours
5073	31–35	14	Non-motorized vehicle accident	135	LOC >30 minutes	epidural hematoma	14	>24 hours
5079	36–40	18	MVA	105	LOC > 30 minutes	PCH; SAH	5	> 24 hours
5082	46–50	12	MVA	90	LOC > 30 minutes	SDH; SAH; bifrontal contusions	14	< 24 hours
5088	31–35	12	Other	66	LOC >30 minutes	N/A	N/A	>24 hours
5095	36–40	12	Other	53	LOC >30 minutes	ICH, parenchymal contusions, SAH, SDH	3	>24 hours
5098	51–55	14	Struck by object	169	LOC <30 minutes	SAH in right frontal lobe and right temporal lobe; right frontal IPH; frontotemporal contusion	N/A	< 24 hours
5100	51–55	18	Other	35	LOC>30 minutes	IPH in left frontal lobe; IVH	3	>24 hours
5104	36–40	20	Struck by object	25	LOC <30 minutes	SDH; scattered SAH; right temporal hemorrhage	15	<24 hours
5109	26–30	14	MVA	106	LOC > 30 minutes	SDH; IPH; IVH	5	>24 hours
5111	18–25	16	MVA	69	LOC <30 minutes	Shear injury; DAI	N/A	>24 hours
5112	51–55	16	MVA	53	LOC >30 minutes	IPH; IVH; Frontal hematoma	10	>24 hours
5115	36–40	12	MVA	205	No LOC	SAH	N/A	>24 hours
5117	46–50	12	MCC	118	LOC <30 minutes	DAI	15	>24 hours
5118	26–30	18	MVA	50	LOC >30 minutes	SDH	10	>24 hours
5119	36–40	16	MVA	226	LOC > 30 minutes	SAH; right frontal contusion	N/A	>24 hours
5122	51–55	18	Non-motorized vehicle accident	23	LOC <30 minutes	SAH	15	>24 hours
5123	51–55	12	MCC	25	LOC <30 minutes	IPH; SDH SAH	14	>24 hours
5124	18–25	12	Fall from height	32	LOC > 30 minutes	IPH; IVH	3	>24 hours
5125	51–55	12	Ground level fall	15	No LOC	SDH; SAH	15	N/A
5126	41–45	12	MVA	26	LOC >30 minutes	SDH	3	>24 hours
5128	36–40	16	MVA	184	LOC >30 minutes	N/A	N/A	>24 hours
5129	51–55	12	Fall from height (Thrown from horse)	9	LOC <30 minutes	SDH; SAH	12	<24 hours
5131	41–45	12	MVA	9	LOC >30 minutes	SDH	12	>24 hours
5132	36–40	16	MVA	7	LOC >30 minutes	SAH	N/A	<24 hours
5133	18–25	12	MCC	21	LOC <30 minutes	SDH; IVH; contusions	15	<24 hours
5134	46–50	16	MCC	7	LOC <30 minutes	IPH	12	<24 hours
5137	26–30	16	Ped vs. auto	8	LOC >30 minutes	Epidural hematoma: SDH; SAH	3	>24 hours
5139	18–25	11	Assault	15	LOC >30 minutes	SDH; SAH	14	<24 hours
5140	18–25	12	Fall from height (Thrown from horse)	9	LOC <30 minutes	SAH	9	< 24 hours
5141	26–30	12	MVA	7	LOC >30 minutes	SDH	13	<24 hours
5144	18–25	12	Non-motorized vehicle accident	12	No LOC	SDH; SAH	12	< 24 hours
5146	51–55	12	Fall from height	16	LOC <30 minutes	ICH; SDH; SAH; IPH	15	>24 hours

Note: ID = participant ID number. Education (Edu) reflects years of highest degree obtained. MVA = motor vehicle accident. MCC includes both motorcycle and snowmobile accidents. Non-motor = non-motorized vehicle accident. Ped vs. auto = participant was hit by car while walking or running. Time since onset (TSO) is presented in months. Loss of consciousness (LOC) is presented in minutes. SDH = subdural hematoma. SAH = subarachnoid hemorrhage. IPH = intraparenchymal hemorrhage. IVH = intraventricular hemorrhage. PCH = parenchymal hemorrhage. Glasgow Coma Scale (GCS) is total score at time of first post-injury measurement. PTA = post-traumatic amnesia. N/A = information was not available.

Stimuli

Basic emotion faces

Participants completed the Karolinksi Directed Emotional Faces (KDEF) static face recognition test (54) of seven facial expressions, including the six basic emotions and neutral expressions. We used the same 28 facial stimuli (4 different actors per emotion) as in Rigon, Turkstra, Mutlu, and Duff (22). The static facial recognition task was chosen over the dynamic task to serve as a more direct comparison with the emoji stimuli which are also static images. Participants were presented one colored photo at a time with seven responses to the right of the photo (fearful, angry, disgusted, happy, neutral, sad, surprised). Response labels were presented in the same order on each trial. Participants selected the response that best described the person’s feeling.

Basic emotion emoji

In the same format as above, participants chose the best-matching emotion label for 28 emoji stimuli depicting basic emotions: anger (Unicode 1f620), disgusted (Unicode 1f616), fearful (Unicode 1f628), happy (Unicode 1f642), sad (Unicode 1f641), surprised (Unicode 1f632), and neutral (Unicode 1f610). In addition to referencing the emotion disgust in its Emojipedia description, the Confounded-face emoji (Unicode 1f61) chosen here has also been associated with and used to represent disgust in previous studies (34–36). As with the facial recognition task, four versions of each emoji emotion were shown, using renderings from the Apple (iOS 14.2), Samsung (Samsung One UI 2.5), Google (Google Android 11.0), and Facebook (Facebook 4.0) platforms.

Social emotion emoji

In the final condition, participants chose the best-matching label for emoji depicting social emotions. A selection of potential social emotions was identified from Hareli and Parkinson’s (47) review of 40 different emotion labels that have been used consistently to describe social emotions in the literature. However, not all of these labels had a clear emoji counterpart. To select the emoji stimuli, we searched keywords for each of the social emotion labels on the Emojipedia website (https://emojipedia.org), identifying 22/40 labels with corresponding emoji. After excluding emotions that included emoji that were used for the basic emotions task and collapsing emotions that shared emoji (e.g., shame and embarrassment were represented by same emoji), we narrowed the field down to 11 unique emoji representing social emotions. We selected a final set of six social emotion emoji that did not overlap in their Emojipedia emotion keyword descriptions: flirting (Unicode 1f60f), confident (Unicode 1f60e), embarrassed (Unicode 1f633), proud (Unicode 1f601), remorseful (Unicode 1f61e), and anxious (Unicode 1f630). Consistent with the basic emotion conditions, neutral (Unicode 1f610) was also included, and depictions from Apple, Samsung, Google, and Facebook platforms were used. All emoji in this final set explicitly mentioned the social emotion label they were chosen to represent in their Emojipedia description. See Figure 1 for example stimuli used for each of the three conditions described above.

Figure 1. Example stimuli for each of the three emotion recognition conditions. In the face condition, each emotion was depicted by four different actors. In the emoji conditions, each emotion was depicted by four different platforms, with iOS 14.2 renderings pictured here. .

Procedure

Study procedures and planned analyses were preregistered on Open Science Framework (https://osf.io/c8kyx) prior to data collection.¹ Participants completed all study tasks remotely using Gorilla Experiment Builder (55); www.gorilla.sc). All participants completed experimental conditions in the following order: (1) basic emotions depicted by faces, (2) basic emotions depicted by emoji, (3) filler task where participants labeled basic emotions depicted by faces wearing surgical masks as part of a separate preregistered study, (4) social emotions depicted by emoji. Presentation of all stimuli within each task were randomized for each participant. All participants completed the study using a laptop or desktop computer. Participants completed the task remotely from their home while supervised on a Zoom conference call with an experimenter who viewed the participants’ screen and gave verbal instructions prior to the task. For each condition, participants were instructed to click on the word that best describes what the person/emoji is feeling. Between each condition, participants rated how confident they were at correctly identifying the emotions on a scale of 1 (Not confident at all) to 5 (Extremely confident).

Analysis

Primary analyses

We used mixed-effect regression to predict the likelihood of correctly labeling an emotion as a function of group (Neurotypical, TBI), condition (basic emotion faces, basic emotion emoji, social emotion emoji), sex (female, male), and their interactions using the glmer() function of the lme4 package (56) in R (R Core Team, 2021) with a logit linking function for binary data. Participant group was dummy coded such that neurotypical group served as the reference. Condition was Helmert contrast coded: Condition contrast 1 compared the likelihood of correctly labeling items in the basic emotion faces condition (2/3) to the average of the basic emotion emoji condition (−1/3) and social emotion emoji condition (−1/3), generating a comparison between face stimuli and emoji stimuli. Condition contrast 2 compared the likelihood of correctly labeling emotions in the basic emotion emoji condition (1/2) to the social emotion emoji condition (−1/2), generating a comparison of emotion type within emoji stimuli. Sex was dummy coded such that female participants served as the reference. We initially attempted to fit the model using a maximal random-effects structure (57) with a random intercept for participant and random by-participant slopes for the effect of stimulus condition. This model failed to converge, so the final model includes only random intercept for participant. To interpret significant coefficients for logit-linked binomial regressions, we used odds ratios. We provide descriptive statistics to compare group differences for individual emotions. To compare participants’ confidence in their emotion labels in each stimulus condition, we used ordered regression to predict confidence ratings (Not confident at all, Slightly confident, Somewhat confident, Fairly confident, Extremely confident), as a function of group (Neurotypical, TBI), condition (basic emotion faces, basic emotion emoji, social emotion emoji) and their interactions using the polr() function of the MASS package. Coding for group and condition were identical as above.

Exploratory analyses

Note, in our preregistration (https://osf.io/enjd8/) we had originally planned to include emotion valence (positive, negative) as a fixed effect in the mixed-effect model. However, we removed this variable from our primary analysis for the following reasons: (1) to reduce number of statistical tests, thereby decreasing likelihood of type 1 errors; (2) emotions are not always easily categorized into a positive/negative binary. For example, although surprise is often categorized as positive, it does not have a clear valence connotation (58), and the Neutral-face emoji can be associated with negative valence (59), possibly reflecting the fact that although neutral facial expressions are common, neutral emoji are rare in CMC (37; 3) for basic emotions, there are more negative than positive emotions. This unbalanced distribution may make it more difficult to distinguish negative emotions relative to positive emotions, contributing to valence effects (60,61; 4) valence effects may be confounded with difficulty (61); and (5) significant effects of valence can be driven by a single emotion (22). To explore valence effects in the current study, we conducted two-sample tests for equality of proportions using the prop.test() function in R to compare whether the proportion or correct label responses differed for positive and negative emotions in each of the three conditions. In additional exploratory analyses, we examine the effect of age on emotion label accuracy and intercorrelations between emotion recognition performance in each of the three conditions using Pearson’s correlation coefficients. The results for these exploratory analyses are provided in supplementary materials.

Results

Effect of stimulus condition on emotion recognition accuracy

First, we examined the effect of participant group and stimulus condition on emotion label accuracy (Figure 2). For labeling basic emotions depicted by faces, neurotypical participants correctly labeled an average of 26.33 items (SD = 1.87), and participants with TBI correctly labeled an average of 26.00 items (SD = 1.60). For labeling basic emotions depicted by emoji, neurotypical participants correctly labeled 22.37 items, on average (SD = 2.11), and participants with TBI correctly labeled 22.65 items, on average (SD = 2.36). For labeling social emotions depicted by emoji, neurotypical participants correctly labeled 21.86 items on average (SD = 4.01), and participants with TBI correctly labeled 19.04 items on average (SD = 4.28).

Figure 2. Proportion of correct emotion labels for each of the three stimulus conditions by group. Points indicate mean performance of individual participants. Bars represent standard error of the mean.

Results of the analysis revealed there was no main effect of group (β = −0.10, z = −0.68, p = 0.50); participants with and without TBI did not significantly differ in their likelihood of correctly labeling emotions. There was a significant effect of condition contrast 1 (faces versus emoji; β = 1.45, z = 8.82, p < 0.001); neurotypical participants were 4.25 times more likely to correctly label emotions depicted by faces than emoji (OR = 4.25, 95% CI [3.08, 5.87]). The effect of condition contrast 2 (basic emoji versus social emoji) was not significant for neurotypical participants (β = 0.19, z = 1.48, p = 0.14); neurotypical participants did not significantly differ in their likelihood of correctly labeling basic emotions compared to social emotions depicted by emoji. There was no two-way interaction between group and condition contrast 1 (β = 0.09, z = 0.37, p = 0.71), indicating that both participants with and without TBI had better emotion recognition for faces than emoji. There was a significant two-way interaction between group and condition contrast 2 (β = 0.45, z = 2.49, p = 0.01). To probe this interaction, we re-ran the model, setting the TBI group as the reference level; this analysis revealed a significant effect of condition contrast 2 for the TBI group (β = 0.64, z = 5.00, p < 0.001). Participants with TBI were 1.90 times more likely to correctly label basic emotions than social emotions depicted by emoji (OR = 1.90, 95% CI [1.48, 2.44]).

Effect of sex on emotion recognition accuracy

Next, we examined the effect of participant sex on emotion label accuracy. There was no main effect of sex (β = 0.15, z = 0.96, p = 0.34); neurotypical male and female participants did not significantly differ in their likelihood of correctly labeling emotions. There was no two-way interaction between participant group and sex (β = −0.24, z = −1.14, p = 0.25), indicating that there was no effect of sex in the TBI group as well. In addition, sex did not interact with condition contrast 1 (β = 0.03, z = −0.13, p = 0.89) or condition contrast 2 (β = −0.16, z = −0.87, p = 0.39). Three-way interactions between group-by-sex-by-condition contrast 1 (β = −0.10, z = −0.30, p = 0.76) and group-by-sex-by-condition contrast 2 (β = 0.30, z = 1.15, p = 0.25) were also both non-significant. Thus, participant sex did not play a significant role in emotion recognition in this sample. Previous work identifying sex differences in static facial emotion recognition report that this difference was driven by females demonstrating better recognition of fearful faces than male participants (22). We provide a breakdown of emotion label accuracy by sex, group, and stimulus condition for each individual emotion type in Supplementary Materials A. In the current study, neurotypical females and neurotypical males correctly labeled fearful faces 69% and 77% of the time, respectively, and female participants with TBI and male participants with TBI correctly labeled fearful faces 72% and 61% of the time, respectively.

Differences in emotion recognition accuracy by emotion type

Next, we examined emotion labeling accuracy for each emotion type by stimulus condition to understand whether certain emotions were driving the effects reported above. Although emotions depicted by faces had higher label accuracy than those depicted by emoji, the face emotion fearful was most commonly mislabeled (Figure 3A). Faces depicting fearful were correctly labeled 73% of the time by neurotypical participants and 66% of the time by participants with TBI. To explore which emotions fearful faces were most commonly mislabeled as, we generated a confusion matrix (Table 2). The neurotypical group labeled fearful faces as disgusted 12% of the time. The TBI group labeled fearful faces as disgusted 14% of the time and as surprised 11% of the time.

Table 2. Confusion Matrix Demonstrating Proportion of Basic Emotions Depicted by Faces Correctly and Incorrectly Labeled by Neurotypical Participants and Participants with TBI.

Figure 3. Proportion of correct labels for each emotion type for each stimulus condition and participant group.

Basic emotions depicted by emoji had lower labeling accuracy than the same basic emotions depicted by faces. This difference was driven by two emoji emotions in both the neurotypical and TBI group: disgusted and fearful (Figure 3B). Emoji depicting disgusted were correctly labeled 39% of the time by neurotypical participants and 34% of the time by participants with TBI. Emoji depicting disgusted were most commonly mislabeled as angry or sad by both neurotypical and TBI groups (Table 3). Emoji depicting fearful were correctly labeled 42% of the time by neurotypical participants and 48% of the time by participants with TBI. Emoji depicting fearful were most commonly mislabeled as surprised by both neurotypical and TBI groups.

Table 3. Confusion Matrix Demonstrating Proportion of Basic Emotions Depicted by Emoji Correctly and Incorrectly Labeled by Neurotypical Participants and Participants with TBI.

Emoji depicting social emotions had reduced labeling accuracy across all emotions (Figure 3C), with the least accurate emotion being anxious for both groups. Emoji depicting anxious were correctly labeled 56% of the time by neurotypical participants and 43% of the time by participants with TBI. Emoji depicting anxious were most commonly mislabeled as remorseful for both neurotypical and TBI groups (Table 4). Although social emotions depicted by emoji often generated mislabels, the intended emotions were reflected in the majority response for both groups.

Table 4. Confusion Matrix Demonstrating Proportion of Social Emotions Depicted by Emoji Correctly and Incorrectly Labeled by Neurotypical Participants and Participants with TBI.

Emotion labeling confidence ratings

Next, we examined whether there were group or condition differences in participants’ confidence in their emotion labels using ordered regression. Participants rated confidence on a 5-point scale (1 = Not confident at all, 5 = Extremely confident). There was no main effect of group (β = −0.15, t = −0.56, p = 0.57); participants with and without TBI did not statistically differ on confidence ratings. There was a significant effect of condition contrast 1 (faces versus emoji; β = 1.32, t = 2.81, p = 0.005); neurotypical participants were less confident rating emotions depicted by emoji than faces. Compared to emoji, face stimuli were associated with an 1.32 times increased likelihood of being rated at a higher confidence level (OR = 1.32, 95% CI[0.45, 2.31]). The effect of condition contrast 2 (basic emoji versus social emoji) was not significant for neurotypical participants (β = 0.62, t = 1.61, p = 0.11); neurotypical participants did not significantly differ in their confidence ratings for basic emotions compared to social emotions depicted by emoji. There was no two-way interaction between group and condition contrast 1 (β = 0.11, t = 0.17, p = 0.87), indicating that participants in both groups were more likely to give higher confidence ratings for faces than emoji. There was no significant interaction between group and condition contrast 2 (β = 0.36, t = 0.66, p = 0.51), indicating a lack of difference in confidence ratings for basic and social emotion emoji in the TBI group as well.

Exploratory analyses

Following the primary analysis, we conducted three exploratory analyses, reporting estimated effect sizes and descriptive statistics. First, given the variability in performance in both participant groups, we asked if participants who performed poorly in emotion recognition of faces also had lower emotion recognition scores for emoji. We found a small positive relationship between label accuracy in the face condition and both the emoji conditions (r[100] = 0.25). Second, we examined whether emotion valence was related to label accuracy and found that participants had better label accuracy for positive than negative emotions in the basic emotion faces (M_pos = 0.99, M_neg = 0.90), basic emotion emoji (M_pos = 0.98, M_neg = 0.68), and social emotion emoji conditions (M_pos = 0.71, M_neg = 0.67). Finally, we examined the relationship between emotion label accuracy and participant age for each of the three conditions. The relationship between age and emotion recognition was inconsistent among conditions, ranging from none (r[100] = −0.13) to a small negative relationship (r[100] = −0.24). Full results for these exploratory analyses are presented in Supplementary Materials B. In addition, we have made raw data available for others wishing to conduct additional exploratory analyses (https://osf.io/enjd8/).

Comparison to prior facial emotion recognition findings in TBI

In contrast to our prediction, we did not find reduced emotion labeling accuracy for participants with TBI relative to neurotypical participants or for male participants relative to female participants. These findings are inconsistent with prior work by Rigon and colleagues (22) who used the same basic emotion face stimuli. Both studies have a comparable sample size (51 participants with TBI in the current study compared to 53 participants with TBI in Rigon et al) and used the same inclusion criteria for diagnosis of moderate-severe TBI. Here, we explore other differences between these two samples that may have contributed to different results.

For labeling basic emotions depicted by faces, neurotypical participants in Rigon et al (22), had an average emotion recognition score of 26.18 (SD = 1.98) and neurotypical participants in the current study had an average emotion recognition score of 26.33 (SD = 1.87). Participants with TBI in the current study performed in line with neurotypical participants with an average recognition score of 26.00 (SD = 1.60) compared to 24.45 (SD = 3.01) in Rigon et al (22). A two-tailed t-test revealed participants with TBI in the current study had significantly better facial emotion recognition of basic emotions than the TBI participants in Rigon et al (22), t(102) = 3.26, p = 0.002, d = 0.64.

Participants with TBI in both samples were in the chronic stage of their injury (greater than 6 months post onset) and received a classification of moderate-severe TBI based on the Mayo classification system (53). However, participants with TBI in Rigon et al (22) had a significantly lower mean injury chronicity (M = 14.94 months, SD = 16.33) than participants in the current study (M = 77.64 months, SD = 78.24 months) t(102) = 5.71, p < 0.001, d = 1.11. In exploring other demographic variables between the two samples, participants with TBI in the current study (M = 38.53 years, SD = 10.60) were significantly younger than participants with TBI in Rigon et al (22), (M = 44.14 years, SD = 14.08), t(102) = 2.29, p = 0.02, d = 0.45. The two samples did not differ in level of education: Participants with TBI in the current study had an average education of 14.84 years (SD = 2.44) compared to 14.81 years (SD = 2.15) in Rigon et al (22), t(102) = 0.07, p = 0.95. Similarly, A chi-square test of independence showed no significant difference in distribution of participant sex between the two samples of participants with TBI, X²(1, N = 104) = 0.04, p = 0.85.

Discussion

The current study represents a first investigation of whether emotion recognition of emoji parallel deficits in emotion recognition of faces in individuals with moderate-severe TBI. In a controlled design, we examined the ability of participants with and without TBI to match photos of faces and emoji to a closed set of basic or social emotion labels. We found no significant difference in emotion recognition in moderate-severe TBI compared to neurotypical peers. However, participants in both groups had lower accuracy labeling emotions depicted by emoji than accuracy for faces. Further, participants with TBI, but not neurotypical participants, had reduced accuracy recognizing social compared to basic emotions depicted by emoji. For basic emotions depicted by faces, participants in both groups had most difficulty recognizing fear, whereas for basic emotions depicted by emoji, participants had most difficulty recognizing fear and disgust. Social emotions depicted by emoji had reduced accuracy overall, with the lowest accuracy for emoji depicting anxious. There was no effect of sex on emotion recognition accuracy. Participants in both groups rated their confidence in their emotion labels higher for emotions depicted by faces than by emoji.

Emotion recognition of faces in TBI

The first aim of this study was to replicate results of Rigon and colleagues (22), who found that participants with TBI had lower facial emotion recognition accuracy of static basic emotions than neurotypical participants and that males had lower facial emotion recognition accuracy than females. We used the same set of 28 stimuli of static human faces depicting basic emotions and compared labeling accuracy in a comparable sample size of participants with and without TBI. In contrast to the prior study, we did not find a deficit in emotion recognition for participants with TBI, and male participants did not have lower emotion recognition scores than females. Although these two studies used the same stimuli with the same stimulus layout, task administration differed between studies and thus, was not a direct replication. Participants in Rigon et al (22) completed the task in-person in the physical presence of an experimenter, on a single computer, whereas participants in the current study completed the task remotely on their own personal computers in the virtual presence of an experimenter, while sharing their screens via Zoom. Average scores for both groups in the present study were highly similar to those of neurotypical participants in Rigon et al (22), which suggests that administration method does not explain the difference in findings.

People with TBI are not uniformly impaired at emotion recognition. In their meta-analysis, Babbage and colleagues (13) estimated that 13–39% of people with moderate-severe TBI have deficits in emotion recognition. Therefore, it is possible that the lack of significant difference in overall emotion recognition between the TBI and neurotypical groups in the current study reflects sampling differences across the heterogeneous TBI population (see 43 for a discussion of risks for sample bias in TBI). Although participants with TBI in both samples were in the chronic stage of their injury (greater than six months since onset) and received a classification of moderate-severe TBI based on the Mayo classification system (53), there were some differences in sample demographics. Participants with TBI in the current study had a longer average time post-injury. Although it is possible that longer time post-injury could be linked to recovered emotion recognition ability, there is evidence that emotion recognition deficits in adults with TBI are stable over time (16). Further, people with moderate-severe TBI generally show no change or even a deterioration in functional levels between 5 and 10 years after brain injury (62). This suggests that increased chronicity may not account for fewer errors in the TBI group in the current study.

Another potential explanation for the better performance by current participants than those in Rigon et al (22) is the difference in age: Participants in the latter study were significantly older than participants in the current study. Although we found no correlation between age and emotion labeling accuracy of faces in the current sample, other studies using the dynamic Emotion Recognition Test (27), which uses gradually morphing facial expressions, have found a decline in facial emotion recognition with increasing age (27–29). We also did not replicate the sex differences reported by Rigon et al (22). Differences in that study were small, so the lack of a difference in our study might reflect sample size. Studies of sex differences in social functioning have produced inconsistent results in adults with and without TBI (24). Therefore, future studies are needed to determine if differences truly exist, are idiosyncratic, or not reliably captured by stimuli. Studies have also rarely included gender, and the intersection of sex, gender, and TBI also merits future study.

Emotion recognition of emoji in TBI

A second aim of this study was to examine whether evidence of lower emotion recognition of faces in TBI extends to difficulties recognizing emotions in the emoji modality. Participants in both groups had lower recognition accuracy for emoji than faces, but there was no significant difference between groups. In the basic emotion emoji condition, errors were primarily in recognizing fear and disgust. In studies of face emotion recognition, fear and surprise are commonly confused, sharing common early signals in dynamic facial expression (e.g., upper lip raising and jaw dropping) (63). Similarly, both disgust and anger are characterized early in facial expression by nose wrinkling (63), and therefore, both facial and emoji stimuli may lead to more recognition errors when depicting less intense representations of these pairs of emotions. Studies of emoji emotionality indicate that current emoji lexicons lack a strong representation of disgust (35,36). Confusion matrices indicated that participants with and without TBI often selected different labels than the ones intended when labeling emotions depicted by emoji and rated their confidence in their emotion labels as lower compared to human faces. This is consistent with prior work demonstrating that a single emoji can represent multiple emotions (30,31). Although facial expressions depicted by emoji are simplified relative to human faces, they may not fall into discrete emotion categories. Indeed, it may be considered a strength of emoji that they represent emotion ambiguously, providing a modality that flexibly and creatively combines with written text to create rich communication contexts.

The final aim of the study was to compare emotion recognition for emoji depicting basic and social emotions. There was no difference in accuracy for basic versus social emotions in neurotypical adults, whereas participants with TBI had significantly lower scores for recognizing social compared to basic emotions. There was no parallel decrease in confidence ratings for labeling of social emotions compared to basic emotions in the TBI group, possibly reflecting a lack of deficit awareness. Social emotions are elicited in social contexts and are dependent on interactions with others and inferences about their mental states (47–49), a context that is often challenging for people with TBI. For example, people with TBI can have difficulty inferring the feelings, intentions, and perspectives of others (1,14). These findings provide new insights into how social cognitive deficits in TBI might manifest as barriers in CMC contexts. In particular, people with TBI may be at increased risk for miscommunication when receiving messages with emoji depicting a range of moods and emotions to convey the sender’s mental state.

Future directions

In the current study, we examined how people with TBI perceived emotion depicted by emoji in isolation, but in CMC contexts, emoji are received in a social context. Thus, an important next direction is to examine how emoji impact message interpretation and the functions of emoji for communication in this population. Emoji serve a variety of communicative functions, including establishing emotional tone; lightening the mood; reducing ambiguity of discourse; softening or strengthening a message; expressing humor; promoting interaction; and maintaining and enhancing social relations (39,64–66). It is unknown if people with TBI use and perceive the communicative functions of emoji in context in similar ways.

Emoji can also change the interpretation of a message, influencing how receivers perceive the sender’s mental state, such as senders shifting the interpretation of neutral messages (e.g., “Hey, please call me”) in a negative or positive direction by adding an upset or happy emoji, respectively (67). In addition, emoji can facilitate the interpretation of non-literal language. When undergraduate students were asked to interpret messages containing indirect meanings (e.g., a person is asked about their grade and responds, “Chemistry is a difficult course”), they were better and quicker at recognizing the indirect meaning when the response contained an emoji (68). Emoji also improved interpretation of sarcastic messages for both younger and older adults (69). Thus, emoji play a communicative role in social interaction. It is an open question whether social cognitive deficits in TBI (e.g., theory of mind, irony, sarcasm) are influenced by emoji. There is evidence from one study that participants with TBI benefit from nonverbal cues from gesture to interpret indirect messages (70). Understanding the extent to which emoji can facilitate the ability of people with TBI to interpret non-literal communication would also provide new insights into the potential of adding emoji to written messages to support social communication in this population.

This study was concerned with how people with TBI perceived emoji, but it is another open question how they use emoji. In addition to serving several communicative functions, emoji play an important role in social participation and even follow certain usage rules. For example, emoji are used more often with friends than strangers and more often in positive than negative contexts (66), as well as being deemed more appropriate in some settings (e.g., text messaging and social media) than others (e.g., e-mail) (65). In fact, in work-related contexts, sending smiley emoji actually led to perceptions of reduced competence and effectiveness (71,72), highlighting the context-dependence of emoji use. Communicating effectively and appropriately in written contexts is an important skill for both social and vocational reintegration after TBI.

Finally, the current study investigated how individuals with TBI perceived emoji through a focused emotion-labeling task, and whether any diminished ability to correctly label emoji translates to social perception challenges. CMC theories suggest that, when nonverbal cues are unavailable or diminished, communicators adapt what cues they use and how they use these cues for effective communication (73). It is possible that individuals with TBI look for and effectively utilize other social cues to supplement their interpretation of emoji that communicate social emotions. Little is known about the extent to which and ways in which individuals with TBI adapt to communication environments with diminished cues or environments where they experience deficiencies in social perception. Furthermore, emoji use may be construed as a communication convention rather than expression of emotion, and communicators may use emoji as part of conventions followed by their group (e.g., using Grinning-face emoji with all their in-group communications to express general affinity), regardless of their ability to correctly interpret them in a reliable fashion. Studies have shown strong cultural differences in emoji use (74), which suggest group-based differences may result from differences in cultural and communicative conventions. Naturalistic and cross-cultural studies can inform our understanding of whether or not, and the extent to which, emoji are used as communication convention.

Conclusion

In this study, participants with TBI did not differ from neurotypical peers in their emotional recognition accuracy. Both participants with and without TBI had reduced accuracy labeling emotions depicted by emoji than faces, and participants with TBI had even poorer accuracy when labeling social emotions compared to basic emotions. Although many emoji depict facial expressions, they are much simpler than human faces, and a single emoji may reflect multiple emotions. Thus, emoji represent a flexible modality of communication, and their interpretation depends on the surrounding context. Emoji are a pervasive component of computer-mediated communication. Thus, studying emoji perception and use after brain injury may provide continued insights into communication barriers or supports in this population. Studying how people with TBI use and integrate information from multiple cues in both face-to-face and computer mediated communication contexts will advance understanding of functional communication and social participation in TBI.

Acknowledgments

This work was supported by NIDCD grants R01 HD071089 a to M.C.D., L.T., & B.M.

Disclosure statement

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Supplementary material

Supplemental data for this article can be accessed online at https://doi.org/10.1080/02699052.2023.2181401

Additional information

Funding

This work was supported by the the National Institute on Deafness and Other Communication Disorders [R01 NIH HD071089].

Notes

1. This preregistration also includes procedures for data collection for a survey task in which participants with TBI generated free-text labels and provided valence and arousal ratings for frequently used emoji, as well as answered questions about their motives and attitudes toward emoji use. This survey was always completed before the current study so that the emotion labels used in the current study did not influence their freely generated perceptions of emoji emotions. The survey shares a common aim of understanding perception of emotions depicted by emoji in TBI, but its results are reported separately.