Facilitated orienting underlies fearful face-enhanced gaze cueing of spatial location

Abstract

Faces provide a platform for non-verbal communication through emotional expression and eye gaze. Fearful facial expressions are salient indicators of potential threat within the environment, which automatically capture observers’ attention. However, the degree to which fearful facial expressions facilitate attention to others’ gaze is unresolved. Given that fearful gaze indicates the location of potential threat, it was hypothesized that fearful gaze facilitates location processing. To test this hypothesis, a gaze cueing study with fearful and neutral faces assessing target localization was conducted. The task consisted of leftward, rightward, and forward/straight gaze trials. The inclusion of forward gaze trials allowed for the isolation of orienting and disengagement components of gaze-directed attention. The results suggest that both neutral and fearful gaze modulates attention through orienting and disengagement components. Fearful gaze, however, resulted in quicker orienting than neutral gaze. Thus, fearful faces enhance gaze cueing of spatial location through facilitated orienting.

Keywords:

• gaze direction
• emotion perception
• social attention
• shared attention
• fear

Public Interest Statement

Eyes are important nonverbal communicators of social information including emotional expression and gaze direction. The results from this study suggest that not only does the direction of another’s gaze influence the location of our attention, but this effect is strengthened when gaze direction is accompanied by a fearful facial expression. In particular, the results suggest that the enhanced attention to fearful gaze is specifically attributed to the initial shift of attention to the location of others’ gaze. In short, the direction of gaze from a fearful face reveals important information to observers about the location of potential threat and observers rapidly shift their own attention to this location for immediate processing of its relevance.

Competing Interests

The author declares no competing interest.

1. Introduction

Faces are important for non-verbal communication within one’s social group. From infancy, there is a strong preference for face relevant stimuli and in particular eye contact (Farroni, Csibra, Simion, & Johnson, 2002; Farroni et al., 2005). Indeed, the eye region of a face plays an important role in the expression of emotion (Darwin, 1872), but also—through the direction of gaze—signals the location of others’ attentional focus (Driver et al., 1999; Friesen & Kingstone, 1998). Information about another’s emotional state and/or attentional focus provides observers with information about salient stimuli within the environment that should potentially be approached or avoided.

Fearful faces are particularly salient environmental cues that signal the presence of potential threat and demand immediate attention from the observer to prepare them for fight-or-flight. Fearful faces have been found to automatically modulate attention both when processed consciously (Pourtois, Grandjean, Sander, & Vuilleumier, 2004) and also when processed nonconsciously (Carlson & Reinke, 2008 Fox, 2002). In particular, fearful faces have been found to facilitate or speed the initial orienting of attention to their location as well as sustain attention to their location by delaying the release, or disengagement, of attention from this location (Carlson & Mujica-Parodi, 2015; Carlson & Reinke, 2010; Carlson, Reinke, LaMontagne, & Habib, 2011). While processing fearful facial expressions, humans reflexively orient their gaze to the eye region (Adolphs et al., 2005) and recent evidence suggests that fearful eyes in isolation are sufficient for the capture and hold of spatial attention (Carlson & Reinke, 2014; Carlson, Torrence, Vander Hyde, 2016). Thus, the eyes of a fearful face seem to be particularly important for the recognition of others’ fear and the ability to adaptively orient attention to fearful facial expressions. Indeed, although enlarged eye whites and dilated pupils directly benefit the expresser by increasing the size of visual field (Susskind et al., 2008), the direction of gaze from a fearful face reveals important information to observers about the location of potential threat.

In general, individuals will reflexively orient their attention toward the location of another’s gaze in a process referred to as joint attention (Driver et al., 1999; Friesen & Kingstone, 1998, 2003; Friesen, Ristic, & Kingstone, 2004; Hietanen, 1999). What one individual attends to and finds important, is often found to be important by other individuals as well. Since emotional expressions signal additional salience or value-related information to observers, it would be expected that joint attention might be stronger for faces of emotional expression—especially for fearful faces where gaze direction signals the location of potential threat. An initial six-experiment study found no evidence for an enhancement of gaze cueing by fearful or any other emotional expressions (Hietanen & Leppänen, 2003).¹ Other studies have reported enhanced gaze cueing for fearful faces, but only in individuals selected for high levels of anxiety (Mathews, Fox, Yiend, & Calder, 2003). On the other hand, more recent studies have found facilitated gaze cueing for fearful faces in unselected individuals when using dynamic expressions (Fichtenholtz, Hopfinger, Graham, Detwiler, & LaBar, 2007; Putman, Hermans, & van Honk, 2006) or when searching for targets with emotional significance (Fichtenholtz et al., 2007; Kuhn & Tipples, 2011; Pecchinenda, Pes, Ferlazzo, & Zoccolotti, 2008). Thus, under certain circumstances, fearful facial expressions do appear to facilitate gaze directed attention. However, the nature and extent of these circumstances are not fully known and there is a clear need for further research on this topic.

As mentioned above, the emotional relevance of the target seems to be one factor that plays a role in the degree to which emotional expressions enhance gaze cueing (Bayliss, Schuch, & Tipper, 2010; Kuhn & Tipples, 2011; Pecchinenda et al., 2008). Given that fearful gaze signals the location of potential threat, another highly relevant target characteristic would be its spatial location. Yet, typically, gaze cueing studies of emotion do not directly asses the ability of gaze direction to facilitate localization, but rather assess the ability of gaze direction to facilitate some type of discrimination (Fichtenholtz et al., 2007; Kuhn & Tipples, 2011; Mathews et al., 2003; Pecchinenda et al., 2008) or facilitate the detection of a target without explicitly identifying its location (Hietanen & Leppänen, 2003; Putman et al., 2006). There is some evidence that dynamic fearful gaze facilitates spatial localization (Graham, Kelland Friesen, Fichtenholtz, & LaBar, 2010). However, gaze cueing with dynamic expressions does not allow for the inclusion of forward/straight gaze trials as these trials are static and processed differently than dynamic gaze trials.

As discussed above, the processes that underlie spatial attention can be divided into an initial orienting/engagement stage where attention is captured as well as a later disengagement stage where attention is released (Posner, 1980; Posner, Inhoff, Friedrich, & Cohen, 1987). By including forward/straight gaze baseline trials, which do not bias observer’s attention to one side of the screen or the other, and serve as a baseline for reaction times, the relative contribution(s) of facilitated orienting (faster responses on valid compared to baseline) and delayed disengagement (slower responses on invalid compared to baseline) can be isolated. The primary aim of this study was to assess the extent to which facilitated orienting and/or delayed disengagement accounts for fearful face enhanced gaze cueing of spatial location. Given the mixed reports in the literature and the uncertainty over the circumstances in which fearful facial expressions facilitate gaze cueing, a secondary aim of the current study was to further explore the extent to which fearful faces enhance gaze cueing and in particular, gaze cueing of spatial location. To test these aims, a gaze cueing study consisting of centrally presented faces with fearful or neutral expressions was used. The gaze of the face was directed either to the left, right, or forward. A target dot appeared on the left or right side of the screen and participants were instructed to locate the dot as quickly as possible. Given the results of prior gaze cueing studies reviewed above, it was hypothesized that target localization would be faster at validly cued locations compared to invalidly cued locations and this effect would be larger for fearful faces.

2. Method

2.1. Participants

Fifty-one introductory psychology students (25 male) between the ages of 18 and 31 (M = 20.16, SD = 2.83) participated in the study. Forty-four reported being right-handed and seven reported being left-handed. Review of the box and whisker plots of the simple effects identified one individual whose reaction time (471.02 ms) was greater than 3 standard deviations from the group mean (M = 373.68, SD = 32.03) in this cell. This individual was considered an outlier and subsequently excluded from data analysis (final N = 50, 25 male; age 18–31, M = 19.96, SD = 2.46; 43 right-handed). The Institutional Review Board at Northern Michigan University approved the study and participants received extra credit for their participation.

2.2. Procedure

The gaze cueing localization task was programmed using E-Prime2 software (Psychology Software Tools, Pittsburg, PA). A 60 Hz 17” LCD computer monitor was used to display the task. Four fearful and neutral (half female) grayscale faces from a standardized database were used as the stimuli (Gur et al., 2002). Using Photoshop, the pupils and iris were removed from the center of the eye and positioned in either the left or right corner of the eye to create leftward and rightward gazing faces, respectively. The original location of the pupil was filled in with eye white from the surrounding sclera. For each facial identity, there was a forward, leftward, and rightward gaze for fearful and neutral expressions.

As displayed in Figure 1, each trial started with a white fixation cue (+) in the center of a black background for 1000 ms. Then, a face subtending 6° × 8.5° of the visual angle² replaced the fixation cue (200 ms). The face was centered on the horizontal (X axis) midline with the eyes of the face positioned on the vertical (Y axis) midline. With the face remaining on the screen, a target dot (0.4° of the visual angle) appeared either 11.25° to the left or right side of the face on the vertical midline (i.e. the same horizontal plane as the eyes) and remained until a response was made. Using an E-Prime response box, participants used the first (left most) button with their right index finger to indicate left-sided targets and used their middle finger on the second button to indicate right-sided targets. On all trials, participants were told to locate the dot (left or right) and to respond as quickly as possible. A blank black screen inter-trial interval concluded each trial (1,000 ms). This trial structure then repeated.

Figure 1. Examples of a fearful face forward gaze trial, fearful face invalid gaze trial, and a neutral face valid gaze trial.

Notes: In the gaze cueing task, each trial started with a centrally presented fixation cue, which was followed by a single face. A target dot then appeared either to the left or right side of the screen and participants responded to the location of the dot as quickly as possible. The face was either fearful or neutral in expression. Gaze was either to the left, to the right, or straight ahead (forward). Directed gaze trials could be congruent with the location of the target dot (valid) or incongruent (invalid).

The task contained 128 valid (gaze direction to the same side of the screen as the target dot; 64 fearful and 64 neutral), 128 invalid (gaze direction to the opposite side of the screen as the target dot; 64 fearful and 64 neutral), and 128 forward gaze trials (64 fearful and 64 neutral) occurring in a unique randomized order for each participant. Faster responses on valid relative to invalid trials were used to index the overall gaze cueing effect.³ In emotional spatial cueing tasks, relative orienting and disengagement differences can be made by comparing across different valid cue types (e.g. fear vs. neutral) and invalid cue types, respectively. By including fearful and neutral forward gaze faces, this experiment was able to measure the absolute degree of orienting and disengagement separately for each expression. Faster responses on valid compared to baseline/forward gaze and slower responses on invalid compared to baseline/forward gaze trials were used to index orienting and disengagement specific components, respectively. This method also allowed for the direct comparison of potential differences in the orienting and disengagement components of gaze cued attention by fearful vs. neutral expressions.⁴

3. Results

Reaction time (RT) data were filtered to exclude incorrect, premature (<150 ms), and delayed (>750 ms) responses (Carlson & Reinke, 2008). After filtering, 97.89% of the original data were available for analysis (1.04% excluded for incorrect responses and 1.07% excluded for reaction time). The effects of expression (fearful vs. neutral) and gaze type (valid vs. invalid vs. forward) on participants’ RTs were assessed with a two-way repeated measures analysis of variance (ANOVA). There was a significant effect of gaze type, F (2,98) = 47.82, p < 0.001, η_p² = 0.49, where Bonnferoni corrected follow-up pairwise comparisons indicate that RTs were faster on valid (M = 354.72 ms) compared to invalid (M = 371.26 ms, SEM = 1.86, p < 0.001) and forward gaze trials (M = 362.32 ms, SEM = 1.69, p < 0.001), while RTs for invalid trials were slower than forward gaze trials (SEM = 1.51, p < 0.001, see Figure 2(a)). The main effect of expression approached significance, F (1,49) = 3.79, p = 0.057, η_p² = 0.07,where fearful expressions (M = 361.95 ms) resulted in marginally faster RTs than neutral expressions (M = 363.59 ms, SEM = 0.84). However, this effect was subsumed and better explained by a significant expression by gaze direction interaction, F (2,98) = 3.71, p < 0.05, η_p² = 0.07. Follow-up Bonnferoni corrected pairwise comparisons reveal similar patterns for fearful and neutral expressions. For both fearful and neutral expressions, valid gaze trials (fearful: M = 352.06 ms and neutral: M = 357.39 ms) were faster than invalid (fearful: M = 371.74 ms, SEM = 1.96 and neutral: M = 370.79 ms, SEM = 2.52, ps < 0.001) as well as forward gaze trials (fearful: M = 362.04 ms, SEM = 1.90, p < 0.001 and neutral: M = 362.60 ms, SEM = 1.96, p = 0.03) and RTs for invalid trials were slower than forward gaze trials (fearful SEM = 1.89, p < 0.001 and neutral SEM = 2.19, p = 0.001). Critically, however, RTs on valid trials were significantly faster for fearful compared to neutral expressions (SEM = 1.27, p < 0.001), while RTs did not differ based on expression for invalid (SEM = 1.90, p = 0.62) and forward gaze trials (SEM = 1.63, p = 0.73, see Figure 2(b)). This pattern of results would suggest that attentional orienting of gaze cueing is facilitated by fearful expressions. To verify this, follow-up paired samples t-tests were run to directly compare the difference between orienting (valid—forward gaze) and disengagement (invalid—forward gaze trials) effects for fearful and neutral expressions. Orienting for fearful faces (−9.98 msΔ) was significantly faster than orienting for neutral faces (−5.22 msΔ, SEM = 1.85, t(49) = −2.67, p = 0.01, see Figure 2c) while disengagement for fearful (9.69 msΔ) and neutral (8.18 msΔ, SEM = 2.71, t(49) = 0.56, p = 0.58) faces did not differ.

Figure 2. (a) For both fearful and neutral expressions, valid trials were faster than invalid and forward gaze trials, while invalid trials were slower than forward gaze trials. (b) Valid gaze trials for fearful faces were faster than valid gaze trials for neutral faces; however, invalid and forward gaze trials did not differ for fearful and neutral expressions. (c) There was a larger orienting effect (valid—forward gaze) for fearful compared to neutral faces, while there was no difference for disengagement (invalid—forward gaze trials). Error bars indicate the 95% confidence interval of the mean difference.

Measurements of accuracy were also analyzed using a two-way repeated measures ANOVA with expression (fearful vs. neutral) and gaze type (valid vs. invalid vs. forward) as the independent variables (using the RT filtering described above). There was a significant main effect of gaze type on accuracy, F (1,49) = 11.74, p < 0.001, η_p² = 0.20. Follow-up Bonnferoni corrected pairwise comparisons indicate that responses to valid gaze (99.4% correct) were more accurate than responses to invalid (98.6% correct, SEM = 0.009, p < 0.001) and forward gaze (99.0% correct, SEM = 0.004, p = 0.04). Responses on forward trials were marginally more accurate than invalid trials (SEM = 0.004, p = 0.08). However, there was no main effect of expression (F (1,49) = 0.39, p = 0.54, η_p² = 0.01) and expression did not interact with gaze type (F (1,49) = 0.61, p = 0.52, η_p² = 0.013) on accuracy of responses.

4. Discussion

There were large gaze cueing effects for both fearful and neutral faces on reaction time that were driven by facilitated orienting and delayed disengagement. Fearful and neutral gaze also resulted in enhanced accuracy at validly cued locations. These findings are consistent with the notion that eye gaze from any facial expression provides important information about salient environmental locations (Driver et al., 1999; Friesen & Kingstone, 1998). Critically, however, the results indicated a greater gaze cueing effect for fearful faces, which was specifically attributed to faster responses on valid trials for fearful compared to neutral faces. Additionally, there was a larger difference in reaction time between valid and forward gaze trials for fearful compared to neutral faces—suggesting that the enhanced gaze cueing of spatial location for fearful facial expressions is driven by facilitated orienting.

The results add to the literature on the modulation of gaze cueing by fearful facial expressions and are consistent with the notion that—under certain circumstance—fearful facial expressions do facilitate gaze cueing. Prior research has shown that fearful gaze facilitates the discrimination of threat vs. nonthreat or pleasant vs. unpleasant stimuli (Fichtenholtz et al., 2007; Kuhn & Tipples, 2011; Pecchinenda et al., 2008). For example, you are more likely to follow the gaze of a fearful expression if you are searching for spiders (Kuhn & Tipples, 2011). These findings make evolutionary sense in that fearful faces indicate the presence of potential threat and it would be important for an observer to discriminate between actual threats and false alarms. On the other hand, discriminations on non-threat-relevant dimensions (e.g. uppercase vs. lowercase words) don’t appear to be enhanced by gaze cueing from fearful faces (Pecchinenda et al., 2008), at least in unselected populations (Mathews et al., 2003). From an evolutionary perspective, this also makes sense as such discriminations provide little benefit for fight-or-flight appraisals and responses. The current results add to literature suggesting that in addition to threat-related discriminations, fearful face-directed gaze facilitates the processing of spatial location and aids in spatial discriminations (Graham et al., 2010). Indeed, it is both important for an individual to know a stimulus’s threat relevance as well as its spatial location. It should be noted that prior research indicates that when attentional resources are actively engaged in task- or goal-relevant behaviors, the influence of emotional salience on spatial attention is less effective (Lien, Taylor, & Ruthruff, 2013). However, in situations when endogenous attention is minimally activated, threat-related stimuli, such as fearful facial expressions and gaze, capture observers’ attention. The results of this experiment provide novel evidence that fearful gaze-facilitated spatial processing is specifically associated with a speeded orienting to the location of potential threat. There was also a delayed disengagement of attention from this location; however, this delay was comparable to that of neutral gaze. Thus, fearful gaze does not seem to sustain attention longer than neutral gaze. On the other hand, fearful gaze speeds the initial shift of attention to this location to allow for immediate stimulus processing.

The amygdala has been implicated in the processing of fearful faces (Adolphs et al., 1999; Breiter et al., 1996; Morris et al., 1996; Whalen et al., 1998) and in particular the eye region of fearful faces (Adolphs et al., 2005; Morris, deBonis, & Dolan, 2002; Whalen et al., 2004). Additionally, research with macaques has found that the amygdala has a preference for averted (relative to forward) gaze (Hoffman, Gothard, Schmid, & Logothetis, 2007) and damage to the human amygdala impairs the ability to use others’ gaze to guide attention (Akiyama et al., 2007). The amygdala also appears to play a more general role in directing attention to fearful faces (Bach, Hurlemann, & Dolan, 2014; Carlson, Cha, Harmon-Jones, Mujica-Parodi, & Hajcak, 2014; Carlson, Cha, & Mujica-Parodi, 2013; Carlson, Reinke, & Habib, 2009; Monk et al., 2008; Vuilleumier, Richardson, Armony, Driver, & Dolan, 2004). Furthermore, neurons in the non-human primate amygdala code for both stimulus salience and spatial location (Peck, Lau, & Salzman, 2013). Thus, the amygdala plays an important role in the neural network for the socioemotional guidance of attention (Carlson et al., 2013; Nummenmaa & Calder, 2009) and evaluates stimulus salience and location. The current finding that fearful gaze facilitates location-based discriminations provides a behavioral complement to this neuroscience research. It should be noted, however, that the current study only used broad left vs. right spatial discriminations. Future research is needed to determine the specificity of fearful gaze-facilitated spatial coding.

As mentioned in the introduction, the eye region of the face plays an important role in the expression of emotion including fear, which is characterized by enlarged eye whites and dilated pupils (Darwin, 1872). Given this characteristic feature of fearful facial expressions, it could be argued that facilitated attentional orienting to fearful gaze is driven by size differences between fearful and neutral eyes rather than the emotional expression of the face per se. Although eye widening has been found to facilitate target detection in expressers and identification of others’ gaze direction in observers, research suggests that increasing eye size does not increase attentional cueing by gaze (Lee, Susskind, & Anderson, 2013). Thus, based on this previous work, it seems unlikely that eye size alone could account for the facilitation in attentional orienting for fearful gaze observed in the current study. Although the results suggest that fearful faces facilitate the orienting of attention to gaze location, it cannot be concluded from the current experiment if this effect is specific to fearful facial expressions or to emotional expressions more broadly. Another potential limitation of the current study is the use of only four facial identities. However, given that (1) facial identity and expression are subserved by distinct brain regions (Haxby, Hoffman, & Gobbini, 2000), (2) facial expressions are thought to be universally expressed (Ekman, Sorenson, & Friesen, 1969) and (3) the faces used in this study were from a standardized database (Gur et al., 2002), it is expected that the current results would generalize to other facial identities. Nevertheless, future research could explore the specificity of this effect to fear and the precise role of eye size in the facilitation of orienting toward fearful/emotional gaze.

In conclusion, this study found that both fearful and neutral expressions produce large gaze cueing effects, which include an initial orienting of attention toward the direction of gaze and a delayed disengagement of attention from this location. Although both fearful and neutral gaze held attention for a comparable amount of time, the results suggest that fearful gaze allows for a quicker orienting of gaze-guided attention, which allows for an immediate processing of this spatial location.

Acknowledgments

I would like to thank the students in the Cognitive × Affective Behavior and Integrative Neuroscience (CABIN) lab at Northern Michigan University, and in particular Robert Torrence, Briana Rasmusson, Sarah Akans, Ian Buentello, Claire O’Brien, and Jake Aday, for assisting in the collection of this data.

Additional information

Funding

Funding. The author received no direct funding for this research.

Notes on contributors

Joshua M. Carlson

Joshua M. Carlson is an Assistant Professor of Psychology and the Director of the Cognitive × Affective Behavior & Integrative Neuroscience (CABIN) lab at Northern Michigan University (NMU) in Marquette, MI. The CABIN lab studies how social and/or affective stimuli such as facial expressions influence behavior and the neural circuitry underlying this influence on behavior. The results from this paper—suggesting that we preferentially and automatically direct our attention to the location of others’ fearful gaze—builds upon the broader goals of the lab, which have recently focused on the allocation of attention to fearful facial expressions and in particular the eye region of fearful faces. The lab is currently researching the role of emotional expression in the presence of multiple gaze cues.

Notes

1. It should be noted that this study did not contain a direct comparison of fearful and neutral expressions.

2. Note that the visual angles reported are estimates based on the participants’ distance of 59 cm from the screen where 1 cm is equal to 1° of the visual angle.

3. Many gaze cueing studies use catch trials to ensure participants are compliant with task instructions and are not adopting alternative strategies when only two location response options are possible (i.e. left and right). For example, it could be argued that participants might focus their attention to the left or right side of the screen and simply respond to the presence or absence of the target. The usage of this type of strategy would in theory cancel out valid vs. invalid differences and minimize the likelihood of detecting a validity effect. It is unclear, however, how one could use this strategy and still show a validity effect. Thus, the inclusion of catch trials to rule out alternative strategies is important for protecting against null effects. However, as reported in the results the validity effect was quite strong and thus not including catch trials in this experiment did not affect the results.

4. Note that if neutral and fearful forward gaze trials are equal, then a common baseline exists across cue types. However, if there are differences between neutral and fearful forward gaze trials (e.g. it has been suggested that fearful/threatening stimuli result in behavioral freezing or slowing of reaction time (Mogg, Holmes, Garner, & Bradley, 2008), which could theoretically result in slower responses on all fearful face trials), then different baselines are needed for each cue type. By including separate fearful and neutral baseline trial types in this experiment, any potential main effects of cue type can be accounted for when deriving indices of orienting and disengagement.