Modulations of emotional attention and spatial attention on human visual cortical activities

Abstract

Background: Spatial attention could enhance the neural activities of attended locations in human visual cortex. As a salient stimulus, emotional image could enhance the responses in amygdala and visual areas. However, few studies examined the interaction of the attentional effects induced by emotional stimuli and spatial cues on the neural responses in visual areas.

Methods: In the present study, we used functional magnetic resonance imaging (fMRI) to examine the combined and separated effects of emotional image and spatial cue on the activities in human visual areas. A revised Posner cueing paradigm was utilized. Each participant viewed a fearful image and a peaceful image simultaneously in left and right visual fields. A spatial cue of two dots was then presented in one of the image positions. In this manner, the attentional effects for emotional image and spatial cue could be isolated and combined.

Results: The results showed that spatial cue enhanced the responses in V4, intraparietal sulcus (IPS) and lateral occipital area (LO), while emotional image could enhance the responses in V3, V4 and LO. Importantly, no significant interactions were found in any of the visual areas.

Conclusion: Our results indicate that the two kinds of attentional modulation might not be affected by each other. These findings shed light on the neural mechanism of the emotional attention.

Keywords:

• emotional attention
• spatial attention
• visual cortex
• fMRI

Introduction

Spatial selective attention means that the perception of a stimulus at the cued location can be improved significantly, as indicated by higher accurate rates and shorter reaction times.^1,2 The cueing paradigm is usually termed Posner cueing paradigm. Using such a paradigm, neurophysiological studies show that neural responses are not only found to be increased when attention was paid to the cued location,^3–5 but also on salient stimuli.^6–9

Previous studies using Posner cuing paradigms found that the neural correlates of spatial selective attention consist of a dorsal frontoparietal attentional network (DAN) and a ventral attentional network (VAN),^10,11 and both of them need the involvement of visual cortical activity.¹² Frontoparietal cortex is shown to be a core structure in cognitive functions such as attentional control and working memory.^13,14 Attentional modulation from the frontoparietal areas to visual cortex is essential for the selective attention while selecting visual locations.^12,15,16 In addition to top-down modulation, salient stimulus could also boost the activities in the visual cortex through an automatic bottom-up process.^17–21

One kind of salient stimuli that can lead to automatic, involuntary attentional bias are images with emotional content, such as fearful faces.²² Such an attentional bias was termed as emotional attention.²² The core neural structure underlying emotion processing is amygdala.^22–25 Amygdala not only monitored the emotional value, but also projected to prefrontal areas and striate and extra-striate visual cortex to regulate appropriate reactions.²⁶ Neural projection from the amygdala to occipital cortex might be involved in improving visual perception of emotionally salient stimuli.¹⁸ In addition, studies also found enhanced functional connectivity between amygdala and inferior temporal cortex while processing emotional stimuli.^27,28

Although spatial attention and emotional attention both could enhance the visual processing and neural activity corresponding to the cued location, they are two kinds of attention with different sources and neural pathways. It is still elusive that whether the two kinds of attentional modulation interact with each other. Behavioral studies utilizing Posner cueing paradigm found that the performance in cued spatial locations could be further improved when the cued stimuli was emotion-laden stimuli.²⁰ Consistently, using functional magnetic resonance imaging (fMRI) methods, researchers have found an additive modulation of spatial attention and emotional attention in amygdala and fusiform face area.^29,30 However, these results did not indicate the interaction effect between the two kinds of attention. Event-related potentials (ERPs) studies found that early ERPs component are sensitive to emotional content, followed by a spatial attention sensitive component.^19,31 These results demonstrated two independent modulations on different ERPs components, suggested no interaction between spatial and emotional modulations. However, when the gaze was adopted as the spatial cue, ERPs study showed an interaction effect of expression and gaze on N1 component.³² In summary, there is no conclusion on the interaction effect of spatial and emotional attention.

Previous studies revealed that spatial attention and emotional attention both influence the activities in human visual cortex. In the present study, we aim to investigate whether these two kinds of attentional modulation interact with each other. In order to address this problem, we utilized a modified Posner cueing paradigm with fMRI scanning. We separately manipulated the spatial attention and emotional attention. Functional MRI analysis may help us observe the isolated effects of emotional attention and spatial attention as well as a combined effect of these two kinds of attention in each visual area.

Material and methods

Participants

In total, 17 naïve healthy individuals (9 female, M=20.0 years old, SD=1.4) participated in our experiment. Participants were from Southwest University, China. All participants were right-handed, with no history of neurological or psychiatric problems. Written informed consents were provided by all participants prior to the study, which was approved by the Institutional Human Participants Review Board, Imaging Center for Brain Research, Southwest University. This study was conducted in accordance with the Declaration of Helsinki. One subject was excluded due to excessive head motion in the scanner, resulting in 16 valid subjects in fMRI data analysis.

Materials

Original images consisted of 20 fearful and 20 peaceful pictures taken from the International Affective Picture System (IAPS). Fearful pictures include threatening contents such as bugs, snakes or skulls. Peaceful pictures include nonthreatening animals such as birds or butterflies. Subjects were asked to rate their fearful feelings of these selected IAPS pictures by a 5-point scale, from number 1 “not fearful at all” to 5 “very fearful”. Based on the averaged rating scores, the 10 lowest scored peaceful pictures and the 10 highest scored fearful pictures were selected and presented pairwise during the experiment. The size of each image was 3.2°×4.2°. The spatial cue was two tiny solid dots with diameters of 0.1° and the distance between them was 0.5°, which were arranged horizontally or vertically. A cross always appeared on the center of the screen as a fixation. The distance from the center of the image or of the spatial cue to the fixation point was 6.3°.

Experimental procedure

As shown in Figure 1, subjects were asked to pay attention to the fixation center during the whole experiment. Each trial started with two simultaneously presented pictures which appeared for 50 ms in both sides of the visual field. One of the pictures was a fearful picture. The other was a peaceful one. Pictures were randomly selected from the fearful and peaceful pictures. After a 250 ms blank interval, a two-dot probe appeared for 100 ms in left or right visual fields. Subjects were asked to press keys to indicate the orientation of the probe (ie, ‘1’ for horizontal orientation and ‘2’ for vertical orientation) as accurately and quickly as possible in a 1,600 ms blank interval. Therefore, each trial lasted for 2 s. Intervals between two sequential trials were pseudorandomly selected from 2, 4 or 6 s, resulting in an average inter-trial interval of 4 s. There were 192 pairwise pictures in total, which was divided into 6 runs. Each run consisted of 32 trials, with a 12-s blank interval at the beginning and another one at the ending of the run. The positions of emotional pictures and probes were counterbalanced in each run, resulting in same trial numbers (48 trials) for following four conditions: left fearful picture & left spatial cue, left fearful picture & right spatial cue, right fearful picture & left spatial cue and right fearful picture & right spatial cue.

Figure 1 Schematic description of the stimulus pattern of one trial in the main experiment. Each trial began with two images in the left and right visual fields. One image was a threatening picture and the other was a neutral picture. After a blank interval, a spatial cue was presented in either the left or the right visual fields. Subjects were required to report the orientation of the spatial cue within 1,600 ms.

Retinotopic imaging

Retinotopic visual areas (V1, V2, V3 and V4) were defined by a standard phase-encoded method,^33,34 in which subjects viewed rotating wedge and expanding ring stimuli that created traveling waves of neural activity in visual cortex. More detailed description of the procedure could be found in previous studies.^35–37 A block-design scan was used to define the regions of interest (ROIs), including object-selective areas (LO) and responsive areas in V1, V2, V3, V4 and IPS. Subjects viewed images of fearful pictures, neutral objects and texture patterns (scrambled fear picture), which had the same size and same location as the stimuli used in the main experiment. Images appeared at a rate of 2 Hz in a block of 12 s, interleaved with a blank block of 12 s. Each image was presented for 300 ms, followed by a 200 ms blank interval. Each block type was repeated 5 times in the scan, which lasted 360 s in total. As a result, each block contained 24 trials, and each condition contained a total of 120 trials in this experiment. Subjects performed a one-back task during scanning.

MRI data acquisition

In the scanner, the stimuli were back-projected via a video projector (refresh rate, 60 Hz; spatial resolution, 1,024×768) onto a translucent screen placed inside the scanner bore. Subjects viewed the stimuli through a mirror mounted within head cap. The viewing distance was 83 cm. Functional MRI data were collected using a 3T Siemens Trio scanner with a 12-channel phase-array coil. BOLD signals were measured with an echo planar imaging sequence (echo time, 30 ms; repetition time, 2,000 ms; field of view, 196×196 mm²; matrix, 64×64; flip angle, 90; slice thickness, 3 mm; gap, 0 mm; number of slices, 33; slice orientation, axial). The bottom slice was positioned at the bottom of the temporal lobes. A high-resolution 3D structural dataset (3D magnetization-prepared rapid acquisition gradient echo; 1×1×1 mm³ resolution) was collected in the same session before the functional imaging started. Subjects underwent two sessions, one for the retinotopic mapping and the other for the main experiment.

MRI data processing and analysis

The anatomical volume of each subject in the retinotopic mapping session was transformed into the AC-PC (anterior commissure–posterior commissure) space and then was inflated using Brain Voyager QX (Brain Innovation). Functional volumes in all the sessions of each subject were preprocessed, including 3D motion correction, linear trend removal and high-pass (0.015 Hz) filtering.³⁸ Head motion was constrained at no more than 3 mm in any scanning session. The images were aligned to the anatomical volume of obtained retinotopic MRI mapping and were also transformed into the AC-PC space. The first 6 seconds of BOLD signals was discarded in order to minimize transient magnetic saturation effects. The ROIs in V1, V2, V3, V4 and IPS were defined as areas that responded more intensely to the textured patterns (scrambled picture) than blank interval (P<10⁻⁸, uncorrected) and confined by the V1/V2/V3/V4/IPS boundaries defined by the retinotopic mapping scan. LO was defined as the area that responded more intensely to the intact images than the scrambled images in the occipital-temporal cortex. The averaged Talairach coordinates were provided in Table 1.

Table 1 Averaged Talairach coordinates for each ROI in each hemisphere

ROIs		x	y	z
Left hemisphere	V1	−11.7	−89.6	−5.45
	V2	−17.1	−87.3	−4.1
	V3	−21.8	−83.1	−3.2
	V4	−27.7	−71.8	−11.9
	LO	−41.0	−70.0	−3.3
	IPS	−24.4	−74.1	28.4
Right hemisphere	V1	10.6	−87.8	−2.7
	V2	15.6	−83.9	−3.3
	V3	20.7	−79.5	−3.7
	V4	26.3	−70.8	−9.8
	LO	40.9	−66.5	−7.2
	IPS	25.9	−73.7	30.5

Note: Data were averaged from all the subjects.

For each visual field (left or right), there were four conditions of stimulus combinations. For example, for the left visual field, in one quarter of trials, the fearful picture was first presented in this visual field and then the spatial cue was also presented here. Similarly, in another quarter of trials, the fearful picture was presented in this visual field but the spatial cue was not presented here. Therefore, for each visual field, the presentation of stimuli could be one of following four situations: spatial cue is congruent to fearful picture (both are presented or not presented); spatial cue is incongruent to fearful picture (only spatial cue or fearful picture is presented). According to the retionotopic projection from visual field to visual cortex, the information of a visual field projects to the contralateral hemisphere. Therefore, for each visual area in each hemisphere, there were also four conditions of stimulus presentation: contralateral to fearful picture & spatial cue, contralateral to fearful picture & no spatial cue, contralateral to neutral picture & spatial cue and contralateral to neutral picture & no spatial cue. We marked these four conditions as E&S (contralateral to emotional picture & spatial cue), E&NS (contralateral to emotional picture & no spatial cue), NE&S (contralateral to neutral picture & spatial cue) and NE&NS (contralateral to neutral picture & no spatial cue). The setting of the conditions is demonstrated in Figure S1A.

The event-related BOLD signals were calculated through custom MATLAB (www.mathworks.com) code separately for each subject, as a method used by Kourtzi and Kanwhisher.³⁹ BOLD signal intensity was first extracted from predefined ROIs in left and right hemispheres of all the subjects by averaging the data from all voxels within the ROIs. Then event-related averaging was applied for each condition and each hemisphere of the ROIs. Next, we averaged the signals of the same condition in both hemispheres. Therefore, we finally obtained the signals for each condition and each ROI of V1/V2/V3/V4/IPS/LO. The BOLD signal change was the normalized BOLD signal relative to the mean intensity of a run in each ROI. For each condition, the activation was defined as the response of the third TR (4–6 seconds) of each trial after stimulus onset, which covered the peak of the hemodynamic response function (HRF) in human visual areas. The procedure of data analysis is demonstrated in Figure S1B.

Experimental design and statistical analysis

In each ROI, there were four conditions: E&S, E&NS, NE&S and NE&NS. The experimental design was a 2 (emotional attention: E/NE) ×2 (spatial attention: S/NS) within subject design. SPSS 16.0 software was used for data analysis. Repeated measures analysis of variance (ANOVA) was performed for each ROI. Paired sample t-tests with Bonferroni adjustment were performed when planned comparisons were needed. The effect size (ES) was provided with each statistical test. The ES was the partial eta squared (η²) for the interaction effect and the main effect in ANOVA. In t-test, the ES was the standardized mean difference (Cohen’s d).

Results

Subjects showed good performance in detecting the orientation of the spatial cue, at a level of nearly or more than 90% accuracy for each subject (range: 89.5–99.8%; M (SD)=96.1% (2.4%)), indicating that they were concentrated on the current task. Then we extracted the BOLD signal change of each condition (E&S, E&NS, NE&S and NE&NS) in each ROI. A 2 (E/NE) ×2 (S/NS) repeated measures ANOVA was used for analysis.

As shown in Table 2 and Figure 2, we did not find significant interaction effect between emotional and spatial attentional modulations in any of the defined ROIs. However, we found that the main effect of emotional cue was significant in V3, V4 and LO (all Ps<0.002), and that the main effect of the spatial cue was significant in V4, IPS and LO (all Ps<0.001). Interestingly, in V4 and LO, we found significant main effects for both emotional and spatial cues, indicating that responses in these areas were both modulated by emotional attention and spatial attention. Yet no significant interaction effects were found in these areas, indicating that the modulations of the two kinds of attention weakly interacted with each other.

Figure 2 Obtained BOLD signal changes in conditions across subjects. Error bars denote one SEM.

Abbreviations: E&S, contralateral to emotional picture & spatial cue; E&NS, contralateral to emotional picture & no spatial cue; NE&S, contralateral to neutral picture & spatial cue; NE&NS, contralateral to neutral picture & no spatial cue; IPS, intraparietal sulcus; LO, lateral occipital.

Table 2 Main effects of emotional attention and spatial attention in ROIs

ROIs	Activation (BOLD signal change)				Main effect
	E&S	E&NS	NE&S	NE&NS	EA			SA
	F(1,15)	P	ES	F(1,15)	P	ES
V1	0.07 (0.02)	0.11 (0.02)	0.08 (0.03)	0.10 (0.02)	0.08	0.779	0.01	3.68	0.074	0.20
V2	0.11 (0.02)	0.12 (0.02)	0.10 (0.02)	0.09 (0.02)	3.02	0.010	0.17	0.19	0.670	0.01
V3	0.14 (0.02)	0.14 (0.02)	0.13 (0.02)	0.11 (0.02)	18.16	0.001	0.55	0.85	0.370	0.05
V4	0.23 (0.02)	0.19 (0.03)	0.22 (0.02)	0.14 (0.01)	14.99	0.002	0.50	58.45	<0.001	0.80
IPS	0.14 (0.02)	0.10 (0.02)	0.14 (0.02)	0.11 (0.01)	0.21	0.653	0.014	41.87	<0.001	0.74
LO	0.21 (0.02)	0.17 (0.03)	0.18 (0.02)	0.12 (0.02)	15.94	0.001	0.52	34.99	<0.001	0.70

Notes: Neural activations were averaged in the same condition across subjects and hemispheres, with standard error of mean (SEM) in parentheses. We performed a 2 (emotional attention vs no emotional attention) ×2 (spatial attention vs no spatial attention) repeated measures ANOVA on the activation in each ROI. F value, P-value and effect size (ES) of the main effects of ANOVA were reported.

Abbreviations: E&S, contralateral to emotional picture & spatial cue; E&NS, contralateral to emotional picture & no spatial cue; NE&S, contralateral to neutral picture & spatial cue; NE&NS, contralateral to neutral picture & no spatial cue; EA, emotional attention; SA, spatial attention; IPS, intraparietal sulcus; LO, lateral occipital.

To further reveal the attentional modulations in each ROI, we compared the activations in each two conditions with paired sample t-test. First, V3 showed a significant main effect of emotional attention. However, we did not find any significant difference between any two conditions (all Ps>0.05, Bonferroni corrected). Second, IPS showed a significant main effect of spatial attention. Consistent to this result, we found that the activation in NE&S condition was higher than that in E&NS condition (t(15)=5.28, P<0.001, Bonferroni corrected, ES=1.32). In addition, E&S condition also showed a higher activation than E&NS (marginally significant, t(15)=2.88, P=0.066, Bonferroni corrected, ES=0.72) and NE&NS (t(15)=3.32, P=0.025, Bonferroni corrected, ES=0.83) conditions. Finally, both V4 and LO showed significant main effects of both kinds of attention. For V4, the activation in NE&S condition was significantly higher than that in NE&NS condition (t(15)=4.80, P<0.001, Bonferroni corrected, ES=1.20). Meanwhile, there was a trend that E&NS condition showed a higher activation than NE&NS condition (t(15)=2.69, P=0.017, uncorrected, ES=0.67). In addition, E&S condition showed a significantly higher activation than NE&NS condition (t(15)=6.99, P<0.001, Bonferroni corrected, ES=1.75). For LO, the results were similar. That is, E&S condition (t(15)=5.32, P<0.001, Bonferroni corrected, ES=1.33) and NE&S condition (t(15)=3.69, P=0.012, Bonferroni corrected, ES=0.92) showed significantly higher activations than NE&NS condition. E&NS condition showed a trend of higher activation than NE&NS condition (t(15)=2.35, P=0.033, uncorrected, ES=0.59). These results indicate that both emotional attention and spatial attention could boost the neural responses in V4 and LO.

We found significant modulations of emotional attention and spatial attention on the neural activities in V4 and LO. Importantly, there were no significant interactions between modulations of emotional attention and spatial attention. Results from ANOVA demonstrated that the two kinds of attentional modulation weakly interacted with each other. We further elucidated it in an exploratory way, by comparing the predicted activation and the observed activation of the two kinds of attention. According to our hypothesis, if the interaction between the two kinds of attention was not evident, we may expect that the modulation of emotional attention and spatial attention (ie, E&S – NE&NS) should be equal to the sum of the modulations of each kind of the attention (ie, (E&NS – NE&NS) + (NE&S – NE&NS)). This exploratory analysis could further indicate whether the emotional attention and the spatial attention may interact with each other through examining whether the effect of the superimposed attention were the sum of the effects of the each kind of attention. After simplifying the equation, we may expect that E&S = E&NS + NE&S – NE&NS. Left of the equation was the observed activation of E&S condition, while right of the equation was the predicted activation of E&S condition. Then we compared the predicted activation and the observed activation of E&S in V4 and LO (Figure 3). Paired sample t-tests showed no significant differences between the observed activities and predicted activities (t(15)=−1.32, P=0.2, ES=0.33 for V4, and t(15)=−0.84, P=0.4, ES=0.21 for LO), which suggested that the prediction was fairly accurate and that the two kinds of attentional modulation hardly interacted with each other.

Figure 3 Results from the exploratory analysis. We compared the observed activation and predicted activation of the E&S (contralateral to emotional picture & spatial cue) condition. The predicted activation was calculated based on the activations of other conditions. The predictions were fairly accurate in both V4 and LO (lateral occipital). Error bars denote one SEM.

Discussion

The present study utilized a spatial cuing paradigm to measure the amplitude modulation in visual cortex, from V1, V2, V3, V4 to IPS and LO. Subjects showed a relative high and stable performance, indicating that they were highly concentrated. The neural activation results showed significant modulation of emotional attention and spatial attention on the activities in visual cortex. Specifically, the main effect of emotional attention was significant in V3, V4 and LO, while the main effect of spatial attention was significant in V4, IPS and LO. Evidently, the activities in V4 and LO were both modulated by the emotional attention and the spatial attention, yet no significant interaction was found in any of the visual areas. Furthermore, the exploratory analysis also revealed that the interaction between modulations of spatial attention and emotional attention on the responses in V4, and LO was not significant.

The finding of significant attentional modulation by spatial cues in visual cortical areas such as V4, LO and IPS is consist with previous findings. Activities in intraparietal sulcus reflect top-down attentional control.^40–42 Damage to it would lead to deficits in visual attention.⁴³ Besides, lateral occipital complex, including V4 and LO,^44–47 could be modulated by spatial attention.^48–50 However, we did not found significant attentional modulation in early visual areas like V1, V2 and V3, which seems different from previous results that showed attentional modulation in primary visual cortex.^51–53 One possible explanation for the negative finding in our study may be the experimental design we adopted. In our study, we used a rapid event-related design, which is limited by relatively lower signal-noise ratio (SNR) in preserving neural activity-induced responses.⁵⁴ Furthermore, Liu et al found that transient attention did not enhance the response in primary visual cortex, which was similar to our results.⁵⁵ Therefore, only strong enough modulation might be observed in the current design. As previous results showed, attentional modulation increased gradually from low-level visual areas to high-level visual areas.^56,57 In another word, the attentional effect in low-level visual cortex is weak and hard to detect. It is reasonable that we only found significant modulation in the mid-level or high-level visual areas of V4 and LO.

Furthermore, our results showed that responses in V3, V4 and LO could be modulated by emotional attention. Indeed, intense emotional stimuli could induce extensive neural responses in visual cortex alone.⁵⁸ Besides, previous EEG studies found that neural activities in the occipital area could be affected by emotional attention.^21,27,59 Here we demonstrated that the emotional attention mainly modulated the activities in the ventral pathway. It should be noted that the emotional attention showed a much weaker effect in IPS than the effect of non-emotional spatial attention, which implied that the emotional attention may not be controlled by the IPS. As we know, amygdala has a close relationship to the emotional attention.²² In addition, evidence shows that the amygdala has strong functional connections with the inferior temporal gyrus,^28,60 and with the LO.⁶¹ These results and our findings together suggested that a subcortical pathway, originating from amygdala, might control emotional modulation. If it is the case, the underlying mechanisms of the emotional attention and spatial attention may be distinct. However, more evidence is needed to directly test the hypothesis.

More importantly, we examined the interaction between the two effects of emotional attention and spatial attention. No significant interaction effect was found in any of the areas including V1/V2/V3/V4/IPS and LO, which indicated that these kinds of attention may modulate the activities independently. This result was consistent to previous findings showing no interaction between the two kinds of attention in FFA.³⁰ To further illustrate the relationship between the modulation of emotional attention and spatial attention, we performed a prediction analysis on the activities in V4 and LO. The results showed that the effect of the superimposed attention was nearly the sum of the effects of the each kind of attention. This finding further indicated that the emotional attention and the spatial attention may not interact with each other, which was consistent with previous EEG studies.^27,62,63

Conclusion

In this study, we found that emotional attention and non-emotional spatial attention both exert influence on the neural activities in the human visual cortex. Importantly, these kinds of attention might have weak interaction with each other, indicating that there might be two independent mechanisms for the emotional attention and the spatial attention.

Acknowledgments

This work was supported by the New Scientist Training Program of Zunyi Meidical University ([2017]5733-014); the Scientific Research Startup Foundation of Zunyi Medical University (F-875; FB-2014-4); the National Natural Science Foundation of China (31400960); and the Guizhou province theory innovation project (Joint project, GZLCLH-2018-094).

Disclosure

The authors report no conflict of interest in this work.

Supplementary material

Figure S1 (A) Demonstration of the setting of conditions. (B) The procedure of fMRI data analysis after preprocessing the functional images.

Abbreviations: E&S, contralateral to emotional picture & spatial cue; E&NS, contralateral to emotional picture & no spatial cue; NE&S, contralateral to neutral picture & spatial cue; NE&NS, contralateral to neutral picture & no spatial cue.