Evaluating the Sensitivity to Virtual Characters Facial Asymmetry in Emotion Synthesis

ABSTRACT

The use of expressive Virtual Characters is an effective complementary means of communication for social networks offering multi-user 3D-chatting environment. In such contexts, the facial expression channel offers a rich medium to translate the ongoing emotions conveyed by the text-based exchanges. However, until recently, only purely symmetric facial expressions have been considered for that purpose. In this article we examine human sensitivity to facial asymmetry in the expression of both basic and complex emotions. The rationale for introducing asymmetry in the display of facial expressions stems from two well-established observations in cognitive neuroscience: first that the expression of basic emotions generally displays a small asymmetry, second that more complex emotions such as ambivalent feeling may reflect in the partial display of different, potentially opposite, emotions on each side of the face. A frequent occurrence of this second case results from the conflict between the truly felt emotion and the one that should be displayed due to social conventions. Our main hypothesis is that a much larger expressive and emotional space can only be automatically synthesized by means of facial asymmetry when modeling emotions with a general Valence-Arousal-Dominance dimensional approach. Besides, we want also to explore the general human sensitivity to the introduction of a small degree of asymmetry into the expression of basic emotions. We conducted an experiment by presenting 64 pairs of static facial expressions, one symmetric and one asymmetric, illustrating eight emotions (three basic and five complex ones) alternatively for a male and a female character. Each emotion was presented four times by swapping the symmetric and asymmetric positions and by mirroring the asymmetrical expression. Participants were asked to grade, on a continuous scale, the correctness of each facial expression with respect to a short definition. Results confirm the potential of introducing facial asymmetry for a subset of the complex emotions. Guidelines are proposed for designers of embodied conversational agent and emotionally reflective avatars.

Introduction

The use of expressive Virtual Characters is an effective complementary means of communication for social networks offering multi-user 3D-chatting environment (Gobron et al. 2012). We examine here the interaction context where users are unable to directly control their avatar facial expression through their own facial expressions (such type of real-time performance animation has already been successfully addressed in Ichim, Bouaziz, and Pauly (2015). Compared to real-live human-human conversations, how to deliver a realistic and effective communication between virtual characters in VR social platform is still an open question (Gratch and Marsella 2005).

Ahn et al. (2012) proposed an emotional model to formulate and visualize the facial expression of virtual humans in a conversational environment by using Valence-Arousal-Dominance (VAD) parameters. This approach has the very interesting degree of freedom of allowing to generate asymmetric facial expression. Indeed, despite a large body of experimental work revealing facial asymmetry during the expression of basic emotions (Borod, Haywood, and Koff 1997), no systematic effort has been done to transfer this human characteristic onto online conversational agents and real-time autonomous virtual humans. We relate this lack to two causes: the limited computing resources during real-time interaction that led to sacrifice the integration of a model of asymmetric facial expression, and the lack of knowledge about our sensitivity to facial asymmetry of virtual characters. One consequence is that purely symmetric faces and emotion display may have contributed to what makes the virtual character look artificial. Conversely, synthetic characters appearing in feature films can nowadays display the full spectrum of human emotions, including complex ones inducing facial asymmetry, because they are finely crafted by animators from the input of directors over as much larger time frame.

Yet, beyond the facial animation level, the automatic control of virtual characters requires a higher level model of emotion also compatible with real-time performance. For this reason we rely on a well-known three-dimensional model allowing the instantaneous expression of emotions driven by the VAD parameters. Within that framework we can automatically map a 3D VAD emotional state to a facial expression (Ahn et al. 2012). In this way, a much larger range of emotions can be synthesized beyond the six basic emotions identified by Ekman (1971) (anger, disgust, fear, happiness, sadness, surprise). This approach also overcomes the significant effort required by the facial morphing technique (Parke and Waters 2008) for which predefined facial expressions have to be designed for each character. However the VAD model cannot produce some complex emotions such as ambivalent feelings that simultaneously integrate more than a single emotion. A frequent occurrence of such context results from the conflict between the truly felt emotion and the one that should be displayed due to social conventions. In such a case, the scalar valence parameter is not sufficient to model the conflict between a negative and a positive emotional context (Norman et al. 2011). Instead it tends to smooth down the two conflicting valences into an average scalar value. Hence, we have proposed a FACS-based system synthesizing complex and ambivalent emotions by combining two distinct VAD emotions for the left and the right sides of the face (Ahn et al. 2013). Likewise the system can also introduce a small degree of asymmetry in the display of basic emotions to produce more ecologically valid expressions (Borod, Haywood, and Koff 1997). A controlled experiment on the human perception of symmetric and asymmetric facial expressions has been conducted and the first results have been presented in Ahn et al. (2013). The results are summarized in Figure 1 which depicts that symmetric facial expression is preferred in basic emotional word expression, regardless of the virtual character genders. However, opposite results have been found in complex emotional word expression, for which the asymmetric facial expression is significantly more effective than asymmetric facial expression.

Figure 1. Summary of the results from (Ahn et al. 2013). Results are the same for both the male (James) and the female characters (Kamila). Basic emotions were rated as less realistic with asymmetry whereas complex emotions were rated as more realistic with degree of asymmetry.

Yet, the extent to which the asymmetrically altered or the bivalent expressions are correctly perceived was not analyzed. Other important questions to examine are whether:

• The side of the asymmetry (left-right vs right-left) induces a different perception of the displayed emotion.

• There exists an interaction between subject gender and character gender.

• Male and female subjects have the same perception in symmetrical and asymmetrical facial expressions.

The paper is organized as follows. The next section reviews past works establishing the asymmetry of human facial expressions and how it has been reflected in the synthesis of real-time facial expression and models of emotions. We then describe the experimental protocol in detail prior to examining how the above-mentioned questions are reflected in the evaluation results. We conclude by providing guidelines for designers of real-time virtual characters and proposing further research directions.

Related works

Asymmetry of human facial expression

Despite the intrinsic symmetry of the human skeleton and face with respect to the medial/sagittal plane, numerous experimental studies have reported with statistical significance that the expression of emotions were more intense on one side of the face [see the survey from Borod et al. (1998)]. It was initially reported in Sackeim, Gur, and Saucy (1978) that emotions are more intense on the left side, and this, independently of the right or left handedness of the subjects. Given the brain organization of motor control, it characterizes a laterality effect with a greater involvement of the right brain hemisphere for emotional expression (Schwartz, Ahern, and Brown 1979). Asymmetry also spreads through the different time scales that are at play in the expression of emotions, from the small timing nuance of a smile (Ekman and Friesen 1982) to the longer lasting emotional coloring that pervades emotional life (Cowie 2009).

Mixed emotions

As recalled in the previous section, Norman et al. (2011) has demonstrated the existence of bivalent state of mind, i.e. the simultaneous presence of emotions with positive and negative valences due to a conflicting situation between avoiding a source of danger and being attracted toward a potential reward. A recent review has confirmed the possibility to experimentally elicit mixed emotions in humans, including happy-sad, fearful-happy, and positive-negative(Berrios, Totterdell, and Kellett 2015). Research on mixed emotions dates back to Kellogg (1915) where mixed emotions were understood as a transient state between opposite affects until one of them dominates. Their duration is sufficient to make such mixed emotions subjectively felt by the subject but also perceived by an external observer. As reported in Berrios, Totterdell, and Kellett (2015) mixed emotions are complex affective experiences that are more than the sum of the individual involved emotions; this makes them difficult to fit in existing dimensional models of emotion (See following section).

Facial expression synthesis

A large body of work has been performed on facial expressivity in general (Parke and Waters 2008) and in real-time interaction with autonomous virtual humans in particular (Vinayagamoorthy et al. 2006). Pelachaud and Poggi provide a rich overview of a large set of expressive means to convey an emotional state (Pelachaud and Poggi 2002) including head orientation. Albrecht et al. (2005) describe a text-to-speech system capable of displaying emotion by radially interpolating key emotions within a 2D emotion space (hence the paper title “mixed feelings” although the proposed approach does not simultaneously integrate two distinct emotions). In 2009 Pelachaud (2009) acknowledges the whole body scale of emotion expression and its temporal organization of multimodal signals. She describes a componential approach where a complex expression is obtained by combining (symmetric) facial areas of source expressions, the final expression resulting from the resolution of potential conflicts induced by the context (e.g. due to social display rules). A study on emotion expression through gaze (Lance and Marsella 2010) stresses the relationship between a three-dimensional emotion model and multiple postural factors including the head and torso inclination and velocity (Patterson, Pollick, and Sanford 2001). The influence of autonomic signals such as blushing, wrinkles, or perspiration in the perception of emotions has been evaluated in de Melo, Kenny, and Gratch (2010). However, the asymmetry is not acknowledged as a determinant factor in these studies.

3D emotion model and facial expression mapping

In the mid 1970’s, a number of psychologists challenged the issue of defining a dimensional model of emotion. An emotion space spanned by three independent dimensions has been proposed with slightly different terms depending on the authors (Averill 1975; Bush 1973; Russell 1980). In the present paper, we adopt the terms Valence, Arousal, and Dominance for these three axes. The Valence axis also referred to as a Pleasure axis represents the positivity or negativity of an emotion. The Arousal axis describes the degree of energy of the emotion. Finally, the Dominance axis indicates the feeling of power carried by the emotion. Based on this 3D emotion space (Ahn et al. 2013) described a linear mapping of an emotion expressed as a 3D point in VAD space to the activity of antagonist facial muscle groups to produce the corresponding facial expression. The identification of this mapping was initiated by exploiting the precious resource of the Affective Norms for English Words (ANEWs)(Bradley and Lang 1999) (this study has experimentally quantified a set of 1034 English words by male and female subjects in terms of their expressed Valence, Arousal, and Dominance). Given a homogenous sampling of eighteen ANEW words over the VAD space, including the six basic emotions, Ahn was able to quantify each muscle group activity as a linear function of the three VAD emotion parameters. Asymmetric facial expressions can then be built either by introducing a small left-right bias for basic emotions, or by combining a different VAD emotion on each side of the face for more complex and ambivalent feelings.

Materials and methods

Participants

We recruited 58 naïve participants (33 M, 25 F), aged (mean 24, standard deviation 2.51, [min:18, max:32]), through the EPFL-UNIL online platform. Each participant received a monetary reward of 10 CHF for 30 minutes. Their degree of familiarity with 3D avatars and real-time characters was ranging from null to very familiar.

Materials

Eight facial expressions encompassing three basic emotions and five ambivalent feelings were produced for both a male and a female 3D virtual character with the VAD-based approach (Ahn et al. 2013). The static images were computed from the 3D models with the same white background as the evaluation screen (Figure 4 right), the same set of virtual lights and the same virtual camera with Unity3D.

Figure 2. Facial expressions for a basic emotion (sadness). This asymmetric facial expression (left) is slightly amplified on the character’s left side of the face. It is built by biasing a single emotion (right).

Figure 3. Facial expressions for a complex emotion (vicious) combining two distinct VAD states for the asymmetric facial expression (left). The symmetric facial expression is obtained by averaging the VAD coordinates of the two asymmetric components (right).

Figure 4. (left) One of the four illustrations of a keyword showing a symmetric and an asymmetric facial expression on the male agent. The other three combinations for this agent and keyword were obtained by swapping the location of the two faces (a) and by mirroring the left and right sides of the asymmetrical face (b). (Right) The stimuli display setup on a 30” screen.

The eight chosen facial expressions were characterized by a single keyword or a short expression, together with an additional short definition to reduce ambiguity. The keywords associated to the basic emotions were sadness, peaceful, and fear while the keywords and expressions used for the more complex feelings were smirk, vicious, “pretend to be cool,” “too good to be true,” and suspicious. Their associated short definitions are provided in Appendix Table 5, while illustrations of all emotions are gathered in Appendix Table 7.

Table 1. Screen location influence (p-value).

	Basic		Complex
	Sym score	Asym score	Sym score	Asym score
p-value	0.449	0.506	0.112	0.267

Table 2. Side of the facial asymmetry influence (p-value).

	Basic		Complex
	Sym score	Asym score	Sym score	Asym score
p-value	0.261	0.958	0.281	0.161

Table 3. Subject gender and character gender (p-value), different scores of symmetry and asymmetry given by male and female subjects.

	James		Kamila
	Sym score	Asym score	Sym score	Asym score
Sadness	0.503	0.846	0.937	0.906
Peaceful	0.841	0.956	0.876	0.580
Fear	0.542	0.579	0.879	0.874
Smirk	0.733	0.552	0.967	0.910
Vicious	0.918	0.915	0.840	0.664
P2bCool	0.868	0.800	0.823	0.856
Good2b	0.736	0.977	0.910	0.664
Suspicious	0.994	0.489	0.988	0.692

Table 4. Subject gender and asymmetry (p-value).

	Male Sym/Asym	Female Sym/Asym
Sadness	<0.001	<0.001
Peaceful	<0.001	<0.001
Fear	0.984	0.800
Smirk	<0.001	<0.001
Vicious	<0.001	<0.001
P2bCool	0.095	0.156
Good2b	0.036	0.003
Suspicious	<0.001	<0.001

Table 5. keywords and the associated short definitions.

[Keyword]	Short definition
[Sadness]	Emotions experienced when not in a state of well-being
[Peaceful]	Free from disturbance
[Fear]	An unpleasant emotion caused by the belief that someone or something is dangerous, likely to cause pain, or a threat
[Smirk]	A forced smile expressing self-satisfaction or disdain rather than pleasure.
[Vicious]	Deliberately cruel or violent, as if enjoying negative feelings
[Pretend to be cool or okay]	Anxious deep inside but try to show off his/her calm
[Too good to be true]	A positive surprise but hard to believe
[Suspicious]	Having or showing a cautious distrust of someone or something

Keywords and short definitions provided for the eight facial expressions in the experiment:

VAD coordinates of asymmetric expressions: As explained before, the basic expressions are built from a single asymmetrically biased reference emotion, whereas complex expressions are built from two distinct emotions. The symmetric expression is given by the average of left and right VAD coordinates. Table 6 indicates the ANEW keywords (indicated in italics) (Bradley and Lang 1999) that were directed used or edited with the VAD-based system from (Ahn et al. 2013):

Table 6. VAD coordinates of ANEW keywords exploited for building the asymmetric expressions. LR indicates the left-biased asymmetric expression.

[Keyword]	ANEW keywords VAD coordinates used for the asymmetric expressions
[Sadness]	LR: sad [−0.85,-0.22,-0.39] with side weights: 0.75 on the left, 0.25 on the right
[Peaceful]	LR: peace [0.68,-0.51,0.11] with side weights: 0.75 on the left, 0.25 on the right
[Fear]	LR: Afraid [−0.75,0.42,-0.56] with side weights: 0.75 on the left, 0.25 on the right
[Smirk]	L: VAD space origin [0,0,0] = neutral, R: happy [1,0.48,0.53] * 1.3
[Vicious]	L: pleasure [0.66,-0.42,0.8] R: jealousy [−0.31,-0.14,0.5]
[Pretend to be cool or okay]	L: Anxious [−0.12,0.46,-0.26] R: relaxed [0.3,-0.54,-0.3]
[Too good to be true]	L: happy [0.5,0.5,0.06] R: sad [−0.59,0.24,-0.56]
[Suspicious]	L: serious [0.3,-0.1,0.1] R: passion [−0.3,0.6,0.6]

The RL is obtained by mirroring the LR expression.

Table 7. One of the four combinations of each of the eight keywords for the male and female agent’s facial expressions. Facial expression stimuli (one of the four possible combinations per character) for the eight emotions:

For a basic emotion such as sadness the asymmetric facial expression (Figure 2 left) is built by biasing the symmetric one (Figure 2 right). Both possible asymmetric biases (left-intense vs. right-intense) are compared to the un-biased symmetric expression. For a complex feeling described by the keyword vicious (Figure 3), the asymmetric facial expression (Figure 3 left) is built by combining two distinct emotions, namely the one corresponding to the ANEW word jealousy on the left side (image-left) and the one corresponding to the ANEW word pleasure on the right side. The symmetric facial expression (Figure 3 right) is obtained from the average VAD value of the two asymmetric components. Both this asymmetric facial expression and its mirror image are compared to the control condition (symmetric). Appendix Table 6 gathers the VAD coordinates of all ANEW keywords used to build the basic and complex emotions.

Figure 4 depicts one of the four illustrations of an emotional keyword for the male character. The other three combinations for this agent were obtained by swapping the location of the two faces [Figure 4(a) left] and by mirroring the left and right sides of the asymmetrical face [Figure 4(b) left]. The illustration also features the instruction to subject (in blue) on top of the screen. Below this line (in red): (1) an index of the current evaluation with respect to the total of 64; (2) a keyword in square brackets; and (3) a short definition with well-known words, were visible below each facial expression, a continuous scale from 0 (= incorrect) to 10 (= correct) was provided with an initial value of 5. The “set score” button at the bottom of the screen allowed to move to the next illustration only after both scores had been edited. Each illustration was presented on a 30” (76 cm) diagonal screen where each facial expression was occupying a surface of 16.6 cm × 13.55 cm /6.5“ × 5.3“ (Figure 4 right).

Experimental design and procedure

Subjects were asked to score both symmetric and asymmetric facial expressions for eight emotional keywords: sadness, peaceful, fear, smirk, vicious, “pretend to be cool,” “too good to be true,” and suspicious. Among these eight keywords, sadness, peaceful, and fear are labeled as basic emotions and others are labeled as complex emotions. We also informed subjects to spend around 25 seconds per question and pay attention to the correspondence of given emotional keyword and conversational agent’s facial expression rather than the graphics rendering of the agent (Figure 4).

For each emotional keyword, both virtual character genders were used for displaying symmetric and asymmetric expressions. Each keyword was presented in the four combinations obtained by swapping the screen location of the two faces (Figure 5a/c and b/d) and by mirroring the left and right sides of the asymmetrical face (Figure 5 a/b and c/d). In total, eight emotional keywords times four combinations for two character genders result in sixty-four questions that were shown in a randomized order. We only enforced an alternation of male /female character gender to minimize the gender influence. A between-trial white screen was displayed during 2 seconds to minimize any influence of the previous trial.

Figure 5. Experimental conditions. The four layout possibilities of an emotion combining the location of the symmetric facial expression either on the left or the right side of the screen (screen location), and the two possible designs of the asymmetric expression (facial asymmetry).

Statistical analysis and results

In the present analysis we examine whether:

• The screen location of the facial expression influences the subject’s decision.

• The side of the asymmetry (left-right vs right-left) induces a different perception of the displayed emotion.

• There exists an interaction between subject gender and character gender.

• Male and female subjects have the same perception in symmetrical and asymmetrical facial expression.

Screen location

First we examine the potential influence of the screen layout. As expected no significant difference was observed (p-values are listed in Appendix Table 1) among ratings of any emotions when presented on the left or the right side of the screen.

Side of the facial asymmetry

Then we examine the potential influence of the face side of the asymmetry. Figure 6 depicts the scores obtained for the two variants of asymmetric expressions for basic emotions and complex emotions. A paired-samples t-test was conducted to compare whether the differences are significant. No significant differences were observed for the present choice of virtual characters and emotions (p-value in Table 2).

Figure 6. Evaluation of basic and complex emotions depending on the two cases of facial asymmetry as depicted in Figure 5.

Subject gender and character gender

We examine here the potential influence of the subject’s gender and the virtual character gender (Male: James, Female: Kamila). For all emotional words, the results of a Welch’s t-test indicate that there is no significant difference of symmetric/asymmetric rating with respect to subject gender or virtual character gender (Figure 7, Appendix Table 3).

Figure 7. Lack of influence of subject’s gender and virtual character gender in the evaluation of the eight emotional words.

Subject gender and asymmetry

Both subject genders consistently evaluate the eight emotional words (Appendix Table 4). The basic emotional words, such as sadness, and peaceful, symmetric facial expression performed significantly better than asymmetric facial expression (p < 0.05). On the contrary for complex word expressions such as smirk, vicious, and suspicious, both male and female subjects gave higher rating to asymmetric facial expression (p < 0.05). For basic emotion word Fear, and complex emotion word P2bCool, both subject genders have the same perception on symmetric and asymmetric expressions, no significant differences exist (p > 0.05). For the complex emotion word Good2b, symmetric facial expression was preferred than asymmetric expression in both Male and Female subjects.

Discussion and conclusion

Based on the results presented in the previous sections, we can provide some refined guidelines compared to Ahn et al. (2013) for designing intuitive user interfaces leveraging on the automated expression of emotions. We first confirmed that the experimental evaluation protocol was not biased by the location of the facial expressions on the screen by showing that this factor had no influence on the ratings. We then investigated the human sensitivity to left and right sides of face in symmetric and asymmetric facial expressions of basic and complex emotions and observed no statistical significance that the expression of emotions was more intensively perceived on one side of the face. This result is useful because, despite the observation gathered on real humans (Borod, Haywood, and Koff 1997), the design of virtual characters does not need to take this factor in consideration for the level of details and lighting conditions that we used. Then we studied whether the gender of both the subjects and the virtual characters had an influence in the evaluation of facial expressions. Likewise the results showed no influence. Finally, when examining in more detail the influence of asymmetry we can refine the guidelines for embodied conversational agents as follows:

• Basic emotions should still be designed with symmetry, otherwise there is a risk of suggesting an ambivalent emotion.

• Asymmetry is successful at conveying only some of the complex emotions. We conjecture that, due to their transient nature (Berrios, Totterdell, and Kellett 2015), mixed emotions may not be optimally conveyed on a static picture. In particular, we observe that complex emotions involving a closed mouth were much better rated than those with a partially open mouth that may appear as less believable when the time dimension is missing.

In the future, further experimental studies should consider: (1) include a greater variety of synthetic characters to reduce confounding factors linked to the numerous character attributes; and (2) integrate a short-term temporal dimension to highlight the transient nature of mixed emotions. We believe the integration of time should re-enforce the expressive strength of asymmetry in more classes of complex expressions, e.g. those including a component of surprise that involves the opening of the mouth.

Acknowledgments

The authors wish to thank Mireille Clavien for the design of the virtual male and female 3D characters, Stéphane Gobron and Quentin Silvestre for their contributions to the Cyberemotions chatting system software architecture, and Dr Arvid Kappas for fruitful discussions on the evaluation of facial asymmetry.

Funding

This work was supported by the EU FP7 CyberEmotions Project (grant 231323).

Additional information

Funding

This work was supported by the EU FP7 CyberEmotions Project (grant 231323).

Appendix