441
Views
13
CrossRef citations to date
0
Altmetric
Original Articles

INFLUENCE OF AUTONOMIC SIGNALS ON PERCEPTION OF EMOTIONS IN EMBODIED AGENTS

, &
Pages 494-509 | Published online: 07 Jul 2010

Abstract

Specific patterns of autonomic activity have been reported when people experience emotions. Typical autonomic signals that change with emotion are wrinkles, blushing, sweating, tearing, and respiration. This article explores whether these signals can also influence the perception of emotion in embodied agents. The article first reviews the literature on specific autonomic signal patterns associated with certain affective states. Next, it proceeds to describe a real-time model for wrinkles, blushing, sweating, tearing, and respiration that is capable of implementing those patterns. Two studies are then described. In the first, subjects compare surprise, sadness, anger, shame, pride, and fear expressed in an agent with or without blushing, wrinkles, sweating, or tears. In the second, subjects compare excitement, relaxation, focus, pain, relief, boredom, anger, fear, panic, disgust, surprise, startle, sadness, and joy expressed in an agent with or without typical respiration patterns. The first study shows a statistically significant positive effect on perception of surprise, sadness, anger, shame, and fear. The second study shows a statistically significant positive effect on perception of excitement, pain, relief, boredom, anger, fear, panic, disgust, and startle. The relevance of these results to artificial intelligence and intelligent virtual agents is discussed.

INTRODUCTION

There is ample empirical evidence that certain emotions can be distinguished in terms of their associated patterns of autonomic nervous system activity (Levenson, Citation2003). These patterns of activation, besides preparing the organism to appropriately respond to the situation that elicited the emotion, communicate the organism's emotional state to others to facilitate an appropriate response to the eliciting situation. This communication is achieved, among others, through wrinkles, blushing, sweating, tearing, and breathing. To the extent that these signals can be realized in embodied agents, which are agents that have virtual bodies and are capable of expressing themselves through their bodies (Gratch et al., Citation2002), they could play a powerful role in facilitating interactions between agents and human users. For instance, communication of emotions in embodied agents has been shown to influence user's politeness (Kramer et al., Citation2003), empathy (Paiva et al., Citation2005), rapport (Gratch et al., Citation2006), trust (Cowell and Stanney, Citation2003), and learning (Lester et al., Citation2000). A challenge then becomes to simulate these autonomically mediated signals so as to improve perception of emotion in embodied agents. This article explores whether simulation of wrinkles, blushing, sweating, tearing, and respiration facilitates the perception of emotions in embodied agents.

A real-time model to simulate wrinkles, blushing, sweating, tearing, and respiration is proposed. Real-time constraints are particularly important in embodied agents because, following the paradigm of human face-to-face interaction, it is necessary to integrate several verbal and nonverbal modalities, and, moreover, these modalities need to synchronize at the subsecond level (Gratch et al., Citation2002). The challenge is thus to strike the right balance between visual realism and behavioral realism. The idea is that if the behavior is simulated properly, the agent need not be photorealistic to promote the audience's suspension of disbelief and therefore provide the illusion of life (Thomas and Johnston, Citation1995). Thus, our approach steers away from expensive physically based models and instead makes use of inexpensive but appropriate computer graphics techniques to simulate the aforementioned autonomic signals. In this regard, the wrinkles, blushing, sweating, and tearing model makes extensive use of the graphics processing unit (GPU). The GPU implements a hardware-supported programmable graphics rendering pipeline where certain stages can be set to run user-defined programs, called shaders, written in a special language (Akenine-Moller et al., Citation2008). The advantage of using the GPU over pure software solutions is the considerable increase in speed we gain from hardware acceleration. In turn, the respiration model uses morphing to animate breathing. Morphing is a technique that, instead of animating the mesh through the skeleton, deforms the vertices directly (Akenine-Moller et al., Citation2008).

Two studies are then conducted to assess the influence of the simulated autonomically mediated signals on perception of emotions in embodied agents. The first study evaluates the influence of the wrinkles, blushing, sweating, and tearing on perception of surprise, sadness, anger, shame, pride, and fear. The study follows a repeated-measures design where subjects compare images of an embodied agent expressing each of the aforementioned emotions with or without wrinkles, blushing, sweating, and tearing. In the second study people are asked to compare videos of the agent expressing affective states with or without respiration. The study focuses on 14 affective states described as having specific respiratory patterns in the literature: excitement, relaxation, focus, pain, relief, boredom, anger, fear, panic, disgust, surprise, startle, sadness, and joy. All respiration patterns are simulated using the proposed model of respiration.

BACKGROUND

Wrinkles

Two kinds of wrinkles can be distinguished (De Graaff, Citation2002): permanent wrinkles, which are caused by aging and habitual facial expressions as the skin looses elasticity, and temporary wrinkles, which are caused by deformations of the skin layers as a result of muscle contraction. In this work we are interested in the subset of the latter that is associated with emotional facial expressions. In particular, the work focuses on simulation of wrinkles in the forehead caused by the expression of surprise, sadness, and anger.

Blushing

Blushing manifests physiologically as a spontaneous reddening of the face, ears, neck, and upper chest as the small blood vessels in the blush region dilate, increasing blood volume in the area (De Graaff, Citation2002). Blushing, aside from being associated with self-consciousness, can be accompanied by social anxiety, uneasiness, embarrassment, shame, or happiness (e.g., when someone receives an exaggerated praise) (Leary et al., Citation1992). Several theories of blushing have been proposed. First, the interpersonal appraisal theory argues that blushing arises from being self-aware and thinking about what others are thinking of us (Harris, Citation1990). Second, the communicative and remedial theory argues that blushing is a save-face action that acknowledges and apologizes for breaking an accepted social rule (Castelfranchi and Poggi, Citation1990); the social blushing theory expands on the previous one (e.g., explaining cases where blushing occurs with positive emotions) and argues that blushing occurs when undesired social attention is given to someone (Leary et al., Citation1992). In this work we are interested in the fact that blushing serves an important communicative function and is associated with certain characteristic emotions. In particular, the work focuses on simulation of blushing associated with two self-conscious emotions: shame (with negative valence) and pride (with positive valence).

Sweating

Sweating is primarily a means of thermoregulation but can also be caused by emotional stress (De Graaff, Citation2002). This latter form is referred to as emotional sweating and manifests physiologically in the palms of the hands, soles of the feet, axillae, and head (Kuno, Citation1956; McGregor, Citation1952). This form of sweating may occur in situations where an individual is subjected to fearful situations or the scrutiny of others (e.g., talking in public or to a superior) and is particularly evident in shy and social phobic individuals (Scheneier et al, Citation1996). This work focuses on the simulation of sweating in the forehead associated with fear.

Crying and Tearing

Crying is usually associated with the experience of intense emotions in situations of personal suffering, separation, loss, failure, anger, guilt, or joy (Miceli and Castelfranchi, Citation2003). Crying manifests physiologically through the shedding of tears and a characteristic noise (which might become concealed with age). Several explanations have been advanced for crying. In one view, crying is seen as being cathartic and a release after an intense experience (Efran and Spangler, Citation1979), whereas in another view, attachment theory explains crying as an appeal for the protective presence of a parent (Nelson, Citation1998). For the infant, crying is used to call the attention of its caretakers in face of some urgent need (e.g., danger). Later, in adulthood, crying continues to be a reaction to a loss and to carry an attachment message that seeks to trigger a response from its “caretakers” (e.g., spouse or friends). Thus, two factors motivate the simulation of tearing in our work: first, the important communicative function it serves and, second, its association with the expression of strong emotions. The focus of the work is in the simulation of tearing that occurs when experiencing intense sadness.

Respiration

Several affective states have been argued to associate with specific respiration patterns. Excitement has been shown to correlate with faster and deeper breathing (Boiten et al., Citation1994; Gomez et al., Citation2005; Boiten, Citation1998). Enhanced respiration in this case is argued to reflect readiness for action such as in preparation for flight or fight and therefore serves an adaptive purpose. In contrast, relaxation is usually associated with slow and deep breathing (Boiten et al., Citation1994; Wientjes, Citation1993). Fast and shallow breathing tends to associate with behavioral demands that may require restrained and goal-directed attention (Boiten et al., Citation1994; Allen et al., Citation1986; Wientjes, Citation1993). Emotionally, this pattern is usually associated with focus or concentration. Pain has been associated to an increase in respiratory volume (i.e., faster and/or deeper breathing), a lengthening of postinspiratory pauses and breath-to-breath variability (Boiten, Citation1998; Feleky, Citation1914). If the increase in respiratory volume can be explained by the aversive nature of a painful task, breath holding is seen as instrumental in reducing the aversive stimulation (Boiten, Citation1998).

More than serving a respiratory need, nonrespiratory movements are generally viewed as the expression of emotional states (De Graaff, Citation2002). Specifically, laughter has been associated with joy (Boiten, Citation1998; Svebak, Citation1975); sighing with relief, boredom, and anxiety (Boiten et al., Citation1994; Clausen, Citation1951); and yawning with drowsiness and boredom (Boiten et al., Citation1994; Provine and Hamernik, Citation1986). Finally, even though there is some evidence that respiration correlates to dimensions rather than categories of emotions (Boiten et al., Citation1994), several researchers have attempted to find respiration patterns specific to basic emotions (Ekman, Citation2003) such as anger, fear, disgust, surprise, sadness, and joy. Anger and fear are usually associated with increased arousal and therefore tend to be associated with the same respiratory pattern of excitement (i.e., faster and deeper breathing). This has been confirmed in many studies (Boiten et al., Citation1994; Ax, Citation1953; Feleky, Citation1914; Rainville et al., Citation2006). Anxiety seems to be associated with a different pattern: fast and shallow breathing (Christie, Citation1935; Suess et al., Citation1980; Bloch et al., Citation1991). Anxiety, in turn, is well known to relate to panic (Roth and Argyle, Citation1988). Disgust was found to correlate with inspiratory pauses (Boiten, Citation1998). Surprise correlates with the typical phasic interruption of breathing that occurs upon presentation of unexpected or novel stimuli (Sokolov, Citation1963; Barry, Citation1982). A variation of surprise is startle, which has a more intricate but similar effect on respiration. When the event that causes startle occurs, if the person is inspiring, inspiration is accelerated and prolonged, but if the person is expiring, expiration ceases to give way to a quick inspiration (Lang et al., Citation1990; Blatz, Citation1925; Keatinge and Nadel, Citation1965). Sadness and joy seem to have a similar respiratory manifestation of slow and deep breathing (Boiten et al., Citation1994; Feleky, Citation1914; Bloch et al., Citation1991). A special case of joy, as was described above, involves laughter, and this has the typical respiration pattern of laughing (Boiten, Citation1998; Svebak, Citation1975). Table summarizes the respiration patterns reviewed here.

TABLE 1 Respiration Patterns Associated with Affect

MODEL

Wrinkles are simulated using bump mapping with normal maps. One normal map represents a typical temporary wrinkle pattern associated with a certain emotion. Wrinkle dynamics are then synchronized with the underlying pseudo-muscular model for facial expressions (de Melo and Paiva, Citation2006). To implement this, three steps are taken. First, the vertex structure is augmented to contain binormal and tangent vectors that, together with the normals in the normal map, define a frame of reference on which lighting calculations, accounting for bump mapping, are performed (Akenine-Moller et al., Citation2008). Second, normal maps for the wrinkle patterns are created. The third and final step is to create a shader program to run in the GPU that, given the data from the previous steps, actually applies the bump mapping technique while at the same time providing the following expression parameters: one or more normal maps to apply and the interpolation level between the images with and without bump mapping applied. The first parameter supports composition of wrinkle patterns, whereas the second implements wrinkle dynamics by synchronizing it with changes in the pseudo-muscular model of the face. The results for the emotions of surprise, anger, and sadness are shown in Figure to (c). Figure shows how this effect can also be applied to simulate bulging of arteries in anger.

FIGURE 1 Expressions of (a) surprise, (b) sadness, and (c and d) anger using wrinkles.

FIGURE 1 Expressions of (a) surprise, (b) sadness, and (c and d) anger using wrinkles.

The basic idea for simulating blushing is having a way to selectively apply a color tint over certain vertices in the agent's mesh (e.g., the vertices in the cheek). To accomplish this, four steps are taken. First, a floating-point value is added to the vertex structure (which already has position, normal, skinning blend weights and texture coordinates). This value provides the foundation for defining custom subsets of the mesh, which we call mesh masks. A coding scheme, not described in the article, is adopted that supports the association of up to eight masks with each vertex. Thus, mesh masks can be overlapping. Second, a tool is developed to support the interactive creation of mesh masks. Once the mask is finished, the tool allows saving the mask in XML format. Having developed the tool, the third step is to use it to define masks for the areas of the face where blushing is to occur. Two masks are created: one for the cheeks and one for the cheeks, forehead, nose, and ears. The fourth and final step is to create a shader program to run in the GPU that tints the vertices in the specified mask. Several parameters are defined for this shader: color of the tint (e.g., reddish for blushing), mask to apply the tint, and fadeoff at the boundary, which defines how far the pixels in the (outside of the) mask boundary get affected by the color tint. Blushing of the cheeks and the full face, which can be associated with shame or pride, are shown in Figure and (c).

FIGURE 2 Expression of shame using blushing.

FIGURE 2 Expression of shame using blushing.

Simulation of tearing (and sweating) consists of modeling the properties of water and its dynamics. Regarding the former, the material properties of water were defined to have a very high specular component, a low diffuse component (e.g., RGB color of [10, 10, 10]), and a null ambient component. The water is then rendered using bump mapping with a normal map of a typical pattern of tears. The normal map's alpha channel is set to a nonzero value in the tearing (sweating) zone and to zero elsewhere. This channel is then used to composite the tears (or sweat) on top of the agent's image. Moreover, the specular component of the eyes is increased to simulate accumulation of water in the eyes in the case of tearing. Regarding dynamics, a dynamics texture is defined, which consists of a unique grayscale texture that defines how tears (or sweat) evolve in time being black the earliest and white the latest. This texture can then be used to interpolate a value that defines how much of the normal map is rendered at each instant. Finally, both the properties of water and its dynamics are defined in a shader program to run in the GPU. Results for the expression of sadness using tears are shown in Figure to (c). Figure shows simulation of sweating in fear.

FIGURE 3 Expressions of sadness using tears and of fear using sweating.

FIGURE 3 Expressions of sadness using tears and of fear using sweating.

Morphing is used to animate respiration. Morphing is a real-time technique where the agent's mesh is deformed by displacing its vertices locally (Akenine-Moller et al., Citation2008). The idea is to create several deformed poses of the mesh, called morph targets, and then animate by interpolating between the neutral mesh and these targets. Moreover, mixed deformations can be created by blending several targets. In our case, the neutral target corresponds to maximum expiration. This should correspond to minimum volume in the lungs. A morph target, named the breathing target, is then defined for maximum inspiration. Anatomically, this corresponds to forced inspiration and should correspond to maximum volume of the lungs. Finally, morphing supports interpolation between these two targets.

Respiration is then visually characterized by the respiration rate and a respiration cycle. The respiration rate defines how often the respiration cycle repeats itself each minute. The respiration cycle defines how the respiratory volume changes with time. Thus, a convenient way of modeling a respiration cycle is as a function that defines the respiratory volume (image) at each instant of time (domain). The image can be constrained to lie in the range [0.0, 1.0], where 0.0 corresponds to the minimum lung volume (i.e., the neutral morph target) and 1.0 to the maximum lung volume (i.e., the breathing target). Regarding the domain, if the values are interpreted as absolute, then the respiratory rate is defined by the total length of the cycle; if the values are interpreted as relative, then the cycle is time-warped to match an intended respiratory rate. Using this abstraction several kinds of respiration cycle functions are supported: linear defines simple linear interpolation back and forth between minimum and maximum values; keyframe supports definition of an arbitrary shape for the cycle by interpolating <time, volume > pairs; and spline supports definition of arbitrary shapes for the cycles according to one-dimensional spline curves (Akenine-Moller et al., Citation2008). Transition between cycles can occur at the end of the current cycle to avoid a discontinuity in the animation, or immediately.

Finally, to build complex respiration cycles and facilitate transitions between cycles, a stack of respiration layers is defined. A respiration layer defines a respiration cycle and the number of repetitions the cycle is to execute (which can be infinite). The respiration animation is defined by the topmost layer. Once the cycle has executed all the repetitions, the layer is popped and the next layer executes its cycle. For instance, default quiet respiration could be set to run forever in the bottom-most layer and a temporary cycle, such as deep and fast breathing for excitement, could be loaded in the topmost layer to run for a finite number of repetitions.

EMPIRICAL EVALUATION

Study 1: Wrinkles, Blushing, Sweating, Tearing, and Emotions

The first study was conducted to evaluate the influence of the wrinkles, blushing, sweating, and tears model on perception of surprise, sadness, anger, shame, pride, and fear. The experiment follows a repeated-measures design with two conditions per emotion: control, where the agent uses only facial expression to convey the emotion, and expressive, where the agent uses facial expression and wrinkles, blushing, sweating, and tears to convey the emotion. Subjects are asked to classify, for each condition, whether the agent expresses the emotion on a scale from 1 (meaning “doesn't express the emotion at all”) to 10 (meaning “perfectly expresses the emotion”). The order of presentation of the emotions is randomized. The order of presentation of the conditions, given an emotion, is also randomized. The agent, in both conditions, assumes a typical muscular configuration of the face (Ekman, Citation2003). The agent in the expressive condition relies, additionally, on wrinkles, blushing, sweating, and tears as follows: Surprise, sadness, and anger are given typical wrinkle patterns in the forehead; sadness is also associated with tears and shiny eyes; anger is also associated with bulging of arteries in the neck region and a light reddening of the face; pride and shame are associated with blushing of the cheeks; and fear is associated with sweating of the forehead. Figure summarizes these conditions.

FIGURE 4 Control and expressive conditions in study 1.

FIGURE 4 Control and expressive conditions in study 1.

The survey was implemented as an online survey. Forty-four participants were recruited with the following age distribution: 11–20 years, 6.8%; 21–30 years, 47.7%; 31–40 years, 31.8%; 41–50 years, 6.8%; and, 51–60 years, 6.8%. Gender distribution was as follows: female, 54.6%; male, 45.4%. Most had college education or above (90.9%) from diverse fields. Participants had diverse origins: North America, 38.6%; Europe, 36.4%; Asia, 13.6%; and Africa, 11.4%.

The Kolmogorov–Smirnov test was applied to assess the normality of the data in each condition in each emotion. The results show that the control conditions for surprise (D(44) = 0.12, p > .05), sadness (D(44) = 0.13, p > .05), and shame (D(44) = 0.10, p > .05) are significantly non-normal. Therefore, the dependent t test was used to compare means between the expressive and control conditions in pride, anger, and fear, and the Wilcoxon signed ranks test was used to compare ranks between the expressive and control conditions in surprise, sadness, and shame. The results are shown in Table .

TABLE 2 Descriptive Statistics for the Control and Expressive Conditions and Significance Levels in the First Study (n = 44)

Study 2: Respiration and Emotions

The second study assesses the influence of respiration on perception of affect in embodied agents. Fourteen affective states are explored: excitement, relaxation, focus, pain, relief, boredom, anger, fear, panic, disgust, surprise, startle, sadness, and joy. The study follows a within-subjects design where subjects compare videos of the embodied agent expressing the affective states with or without specific respiration patterns. Thus, for each affective state there are two conditions: control, the agent expresses the affective state using neutral respiration and appropriate postures and facial displays, and expressive, the agent expresses the affective state using the same postures and facial displays as in the control condition, plus specific respiration patterns. Subjects are asked to classify, for each condition, how well does the embodied agent appear to be experiencing the affective state from 1 (meaning “not experiencing the affective state at all”) to 10 (meaning “perfectly experiencing the affective state”). The hypotheses are that subjects will perceive, for each affective state, the embodied agent to be experiencing the affective state more intensely in the expressive than in the control condition.

In the control condition the embodied agent expresses the affective state using postures, facial displays, and neutral respiration: 15 breaths per minute and respiration depth ranges from 5% to ∼25% of the maximum respiratory depth (De Graaff, Citation2002). In the expressive condition the embodied agent expresses the affective state using the same postures and facial displays as in the control condition, plus the specific respiration patterns described in the Background section. In both conditions the agent is standing up with the arms hanging to the sides. Idle motion is applied consisting of blinking and very subtle movement of the neck, spine, arms, and fingers. The camera is focused on the chest framing the upper body at a 45-degree angle to the right with respect to the vertical axis. Finally, the agent is illuminated using a strong white light from the front and a dim white light from the right. The video showing all conditions follows with the supplemental materials and can be found at the following website: http://www.youtube.com/watch?v=pAVND3GPKA4.

The survey was anonymous and implemented as an online survey. Forty-one participants were recruited with the following age distribution: 11–20 years, 4.9%; 21–30 years, 56.1%; 31–40 years, 26.8%; 41–50 years, 5.0%; 51–60 years, 4.9%; and, more than 70, 2.3%. Gender distribution was as follows: female, 46.3%; male, 53.7%. The expected or awarded education degree distribution was as follows: basic, 4.9%; college, 24.4%; Masters, 36.6%; and PhD or above, 34.1%. Education majors were in diverse areas. Participants also had diverse origins: America, 46.3%; Europe, 29.3%; Asia, 9.7%; and Africa, 9.7%.

Table summarizes the results for this study. The dependent t test was used to compare means between the expressive and control conditions and calculate significance levels.

TABLE 3 Descriptive Statistics for the Control and Expressive Conditions and Significance Levels in the Second Study (n = 44)

DISCUSSION

The first study shows a large effect of wrinkles, blushing, sweating, and tearing on the perception of anger, sadness, and shame; a medium effect on the perception of fear and surprise; and no effect on the perception of pride. The positive effects suggest that the communicative functions wrinkles, blushing, sweating, and tears serve in human–human interactions (Levenson, Citation2003; Leary et al., Citation1992; McGregor, Citation1952; Miceli and Castelfranchi, Citation2003) also carry to human–agent interactions. That no effect is achieved in pride might suggest that further contextual cues are necessary for associating blushing with pride (Leary et al., Citation1992).

The results of the second study show (a) a large effect of respiration on perception of excitement, pain, relief, anger, fear, and panic; (b) a medium effect on perception of boredom and startle; (c) no effect on perception of relaxation, disgust, surprise, sadness, and joy; and (d) a medium effect against the experimental hypothesis for the perception of focus. The positive results in (a) and (b) suggest that people can, in fact, perceive these affective states more clearly if the respiration patterns that are typically associated with them in people are used. The fact that no effect was achieved for certain affective states can have multiple explanations. First, the videos might have been too short to perceive the influence of respiration for some affective states. This seems to be the case for relaxation and sadness, both of which are expressed by a slow and deep respiration pattern. Second, the absence of effect in boredom and surprise might be explained by imperfections in the experimental setup. In fact, in both cases the associated respiration pattern relies on breathing suspension, and this might have not been clear due to simultaneous secondary motion (e.g., Perlin noise applied to the back and arms). Respiration in laughter also failed to have an effect on perception of joy. This could have been because our simulation of laughter was imperfect. In fact, we simulated laughter by a deep inspiration followed by synchronized convulsions of the chest. However, an in-depth look at the mechanics of laughing (Fillipelli et al., Citation2001) clarifies that convulsions actually occur in the abdominal area. This can be supported in our model by using another morph target that emphasizes motion of the abdomen. Finally, not only did our results fail to support the expressive condition for focus, they actually supported the control condition. This might suggest that our simulation of the typical “fast and shallow” respiration pattern for focus or concentration (Boiten et al., Citation1994) was incorrect. In fact, the literature is not very clear about how much faster and shallower should respiration be and our assumptions on that regard might have been incorrect.

The results also suggest that wrinkles, blushing, sweating, tearing, and respiration can be used to convey intensity of emotion. Effectively, in the first study, even in the control condition, where the agent relied only on proper configuration of the muscles in the face to convey emotion, subjects, on average, were already giving relatively high classifications (surprise, 5.36; sadness, 6.18; anger, 6.02; shame, 5.52; pride, 5.91; fear, 6.36). Still, the expressive conditions managed to increase the average classification for all emotions (surprise, 5.84; sadness, 7.93; anger, 7.91; shame, 6.55; fear, 6.87) but pride (5.73). In the second study, again, though the control conditions got relatively high scores, respiration managed to improve these scores further. The ability to control the intensity of the expressed emotion is, of course, useful in regulating human–agent interactions (Gratch et al., Citation2002). However, display of strong emotions should be used with care. For instance, it would be unreasonable to always use bulging of arteries to express anger or tearing to express sadness because these occur only in specific cases in human–human interactions (Levenson, Citation2003; Nelson, Citation1998).

This article is a step further in clarifying whether autonomic specificity exists in expression of emotion in embodied agents. James (Citation1894) proposed that, in people, emotions could be distinguished in terms of their associated patterns of autonomic nervous system activity. Even though this matter is far from being settled (Cacioppo et al., Citation2000), evidence is growing that autonomic specificity in people exists at least for some emotions (Levenson, Citation2003). Clarifying the autonomic specificity issue is relevant for artificial intelligence because it defines whether an unambiguous mapping exists between affective states and autonomic signals and, if so, what that mapping is. Definition of this mapping, in turn, is relevant to effectively and naturally convey emotion in interactions between embodied agents and people. This article provides preliminary evidence that wrinkles, blushing, sweating, tearing, and respiration can be used to facilitate perception of certain affective states in embodied agents. However, much work still needs to be done.

First, this work does not focus on the dynamics of wrinkles, blushing, sweating, or tearing. In fact, the first study only asks people to compare static images of agents expressing emotions using these signals. Still, it has been shown that subtle variations in facial dynamics can have an impact on how people perceive emotions (Krumhuber et al., Citation2007). What is necessary is another study that explores this issue and asks participants to compare videos of agents expressing emotions using these signals. Second, though we are confident the current model is capable of addressing the issues found in the simulation of some of the respiration patterns, as discussed above, the corrected patterns should be validated with a new sample. Third, it is possible to simulate more autonomically mediated signals so as to convey better or more affective states. For instance, the wrinkles model can be used to simulate bulging of arteries in regions other than the neck and, thus, possibly support the display of more emotions. The blushing model can be used to simulate blanching, pallor, and flushing of the face, which are known to be associated with the occurrence of emotions (Drummond, Citation1998). The sweat model can be used to simulate perspiration in other regions from the body, aside from the forehead. Fourth, whereas this work focuses on respiration patterns as characterized by respiration rate, depth, curve, and nonrespiratory movements, there is also evidence that, in people, affect influences more respiration parameters such as thoracic-abdominal dominance (i.e., whether respiration uses predominantly the abdominal or thoracic muscles; Boiten et al., Citation1994), regularity of breathing (Hormbrey et al., Citation1976), and the inspiration to expiration times ratio (Feleky, Citation1914). Finally, we successfully improved the perception of several affective states using computer graphics techniques, which are simpler than state-of-the-art simulations of these autonomic signals. We are confident that photorealism is not necessary to convey emotions effectively and achieve, with embodied agents, the effect emotions have in human–human interactions. However, to provide evidence for this argument, it is necessary to compare emotion recognition rate when embodied agents express emotions using the current model versus a state-of-the-art anatomical and physically based model of these signals.

Supplemental material

uaai_a_492159_sup_14275333.zip

Download Zip (11.8 MB)

uaai_a_492159_sup_14275433.zip

Download Zip (5.2 MB)

Acknowledgments

This work was sponsored by the U.S. Army Research, Development, and Engineering Command and the NSF under grant HS-0713603. The content does not necessarily reflect the position or the policy of the Government, and no official endorsement should be inferred.

Notes

a Effect sizes for the dependent t test and Wilcoxon signed-rank test are calculated as suggested by Rosenthal (Citation1991). The guidelines for interpretation are as follows (Cohen Citation1988): r = 0.10 (small effect), the effect explains 1% of the total variance; r = 0.30 (medium effect), the effect explains 9% of the total variance; r = 0.50 (large effect), the effect explains 25% of the total variance.

*Significant difference, p < .05.

*Significant difference, p < .05.

REFERENCES

  • Akenine-Moller , T. , E. Haines , and N. Hoffman . 2008 . Real-time Rendering . 3rd ed. Natick , MA : A. K. Peters
  • Allen , M. , A. Sherwood , and P. Obrist . 1986 . Interactions of respiratory and cardiovascular adjustments to behavioral stressors . Psychophysiology , 23 : 532 – 541 .
  • Ax , A. 1953 . The physiological differentiation between fear and anger in humans . Psychosom. Med. , 15 : 433 – 442 .
  • Barry , R. 1982 . Novelty and significance effects in the fractionation of phasic OR measures: A synthesis with traditional OR theory . Psychophysiology , 19 : 28 – 35 .
  • Blatz , W. 1925 . The cardiac respiratory, and electrical phenomena involved in the emotion of fear . J. Exp. Psychol. , 8 : 109 – 132 .
  • Bloch , S. , M. Lemeignan , and N. Aguilerat . 1991 . Specific respiratory patterns distinguish among human basic emotions . Int. J. Psychophysiol. , 11 : 141 – 154 .
  • Boiten , F. , N. Frijda , and C. Wientjes . 1994 . Emotions and respiratory patterns: Review and critical analysis . Int. Journal of Psychophysiology , 17 : 103 – 128 .
  • Boiten , F. 1998 . The effects of emotional behavior on components of the respiratory cycle . Biological Psychology , 49 : 29 – 51 .
  • Cacioppo , J. , G. Berntson , J. Larsen , K. Poehlmann , and T. Ito . 2000 . The psychophysiology of emotion . In: The Handbook of Emotion , , 2nd ed. , eds. M. Lewis , and J. M. Haviland-Jones , 173 – 191 . New York : Guilford Press .
  • Castelfranchi , C. , and I. Poggi . 1990 . Blushing as a discourse: Was Darwin wrong? In: Shyness and Embarrassment: Perspectives from Social Psychology , ed. W. R. Crozier , 230 – 254 . Cambridge University Press .
  • Christie , R. 1935 . Some types of respiration in the neuroses . Quart. J. Med. , 4 : 427 – 432 .
  • Clausen , J. 1951 . Respiration movement in normal, neurotic and psychotic subjects . Acta Psychiat. Neurol. Suppl. 68 .
  • Cohen , J. 1988 . Statistical Power Analysis for the Behavioural Sciences , , 2nd ed. Hillsdale , NJ : Lawrence Erlbaun Associates Inc.
  • Cowell , A. , and K. Stanney . 2003 . Embodiement and interaction guidelines for designing credible, trustworthy embodied conversational agents. Proceedings of the International Conference on Intelligent Virtual Agents 2003. Berlin: Springer-Verlag.
  • De Graaff , K. 2002 . Human Snatomy , , 6th ed. New York : McGraw Hill .
  • de Melo , C. , and A. Paiva . 2006 . Multimodal expression in virtual humans. Proceedings of the Computer Animation and Social Agents 2006 (CASA'06) Conference and Computer Animation and Virtual Worlds, 17(3–4):239–248. Geneva, Switzerland, July 5–7.
  • Drummond , P. 1998 . Correlates of facial flushing and pallor in anger-provoking situations . Personality and Individual Differences , 23 ( 4 ): 575 – 582 .
  • Efran , J. , and T. Spangler . 1979 . Why grown-ups cry: A two-factor theory and evidence from the miracle worker . Motivation and Emotion , 3 ( 1 ): 63 – 72 .
  • Ekman , P. 2003 . Facial expressions . In: Handbook of Cognition and Emotion , eds. T. Dalgleish and M. Power . New York : John Wiley & Sons .
  • Feleky , A. 1914. The influence of the emotions on respiration. Psychol. Rev. , 1:218–241.
  • Fillipelli , M. , R. Pellegrino , I. Iandelli , G. Misuri , J. Rodarte , R. Duranti , V. Brusasco , and G. Scano . 2001 . Respiratory dynamics during laughter . Journal of Applied Physiology , 90 ( 4 ): 1441 – 1446 .
  • Gomez , P. , P. Zimmermann , S. Guttormsen-Schar , and B. Danuser . 2005 . Respiratory responses associated with affective processing of film stimuli . Biolog. Psych. , 68 : 223 – 235 .
  • Gratch , J. , J. Rickel , E. Andre , N. Badler , J. Cassell , and E. Petajan . 2002 . Creating interactive virtual humans: Some assembly required . IEEE Intellig. Systems , 17 ( 4 ): 54 – 63 .
  • Gratch , J. , A. Okhmatovskaia , F. Lamothe , S. Marsella , M. Morales , et al. 2006 . Virtual rapport. Proceedings of the 6th International Conference on Intelligent Virtual Agents 2006. Berlin: Springer-Verlag.
  • Harris , P. 1990 . Shyness and embarrassment in psychological theory and ordinary language . In: Shyness and Embarrassment: Perspectives from Social Psychology , ed. W. R. Crozier , 59 – 86 . New York : Cambridge University Press .
  • Hormbrey , J. , C. Patil , M. Jacobi , and K. Saunders . 1976 . Hyperventilation syndrome—analysis of irregularity of breathing . Bull. Eur. Physiopathol. Respirat. , 22 : 518 .
  • James , W. 1894 . The physical bases of emotion . Psychol. Rev. , 1 : 516 – 529 .
  • Kuno , Y. 1956 . Human Perspiration . Spring Field , IL : Charles C. Thomas
  • Keatinge , W. , and J. Nadel . 1965 . Immediate respiratory response to sudden cooling of the skin . J. Appl. Physiol. , 20 : 65 – 69 .
  • Kramer , N. , B. Tietz , and G. Bente . 2003 . Effects of embodied interface agents and their gestural activity. Proceedings of the Intelligent Virtual Agents Conference 2003. Berlin: Springer-Verlag.
  • Krumhuber , E. , A. Manstead , D. Cosker , D. Marshall , P. Rosin , and A. Kappas . 2007 . Facial dynamics as indicators of trustworthiness and cooperative behavior . Emotion , 7 ( 4 ): 730 – 735 .
  • Lang , P. , M. Bradley , and B. Cuthbert . 1990 . Emotion, attention, and the startle reflex . Psychol. Rev. , 97 : 377 – 395 .
  • Leary , M. , T. Britt , W. Cutlip , and J. Templeton . 1992 . Social blushing . Psychological Bulletin , 112 ( 3 ): 446 – 460 .
  • Lester , J. C. , S. G. Towns , C. B. Callaway , J. L. Voerman , and P. J. FitzGerald . 2000 . Deictic and emotive communication in animated pedagogical agents . In Embodied Conversational Agents , ed. J. Cassell , S. Prevost , J. Sullivan , and E. Churchill , 123 – 154 . Cambridge , MA : MIT Press .
  • Levenson , R. 2003 . Autonomic specificity and emotion . In: Handbook of Affective Sciences , eds. R. J. Davidson , K. R. Scherer and H. H. Goldsmith , 212 – 224 . Oxford , NY : Oxford University Press .
  • McGregor , I. 1952 . The sweating reactions of the forehead . Journal of Physiology 116 : 26 – 34 . Oxford, New York: Oxford University Press.
  • Miceli , M. , and C. Castelfranchi . 2003 . Crying: Discussing its basic reasons and uses . New Ideas in Psychology , 21 : 247 – 273 .
  • Nelson , J. 1998 . The meaning of crying based on attachment theory . Clinical Social Work Journal , 26 ( 1 ): 9 – 22 .
  • Paiva , A. , J. Dias , and R. Aylett . 2005 . Learning by feeling: Evoking empathy with synthetic characters . Applied Artificial Intelligence , 19 ( 3-4 ): 235 – 266 .
  • Provine , R. , and H. Hamernik . 1986 . Yawning: Effects of stimulus interest . Bull. Psychonomic Sot. , 24 : 437 – 438 .
  • Rainville , P. , A. Bechara , N. Nawvi , and A. Damasio . 2006 . Basic emotions are associated with distinct patterns of cardiorespiratory activity . Int. J. Psychophysiol. , 61 : 5 – 18 .
  • Rosenthal , R. 1991 . Meta-Analytic Procedures for Social Research (revised) . Newbury Park , CA : Sage .
  • Roth , M. , and N. Argyle . 1988 . Anxiety, panic and phobic disorders: An overview . J. Psychiatr. Res. , 22 ( 1 ): 33 – 54 .
  • et al. 1996 . Social phobia . In DSM-IV Sourcebook , Vol. 2 , 507 – 548 . Washington , DC : American Psychiatric Association .
  • Sokolov , J. 1963 . Perception and the Conditioned Reflex . New York : The Macmillan Company
  • Suess , W. , A. Alexander , D. Smith , H. Sweeney , and R. Marion . 1980 . The effects of psychological stress on respiration: A preliminary study of anxiety and hyperventilation . Psychophysiology , 17 : 535 – 540 .
  • Svebak , S. 1975 . Respiratory patterns as predictors of laughter . Psychophysiology , 12 : 62 – 65 .
  • Thomas , F. , and O. Johnston . 1995 . The Illusion of Life: Disney Animation . New York : Walt Disney Productions
  • Wientjes , C. 1993 . Psychological influences upon breathing: Situational and dispositional aspects. Doctoral Thesis. Catholic University of Tilburg.

Reprints and Corporate Permissions

Please note: Selecting permissions does not provide access to the full text of the article, please see our help page How do I view content?

To request a reprint or corporate permissions for this article, please click on the relevant link below:

Academic Permissions

Please note: Selecting permissions does not provide access to the full text of the article, please see our help page How do I view content?

Obtain permissions instantly via Rightslink by clicking on the button below:

If you are unable to obtain permissions via Rightslink, please complete and submit this Permissions form. For more information, please visit our Permissions help page.