Modelling audiovisual integration of affect from videos and music

ABSTRACT

Two experiments examined how affective values from visual and auditory modalities are integrated. Experiment 1 paired music and videos drawn from three levels of valence while holding arousal constant. Experiment 2 included a parallel combination of three levels of arousal while holding valence constant. In each experiment, participants rated their affective states after unimodal and multimodal presentations. Experiment 1 revealed a congruency effect in which stimulus combinations of the same extreme valence resulted in more extreme state ratings than component stimuli presented in isolation. An interaction between music and video valence reflected the greater influence of negative affect. Video valence was found to have a significantly greater effect on combined ratings than music valence. The pattern of data was explained by a five parameter differential weight averaging model that attributed greater weight to the visual modality and increased weight with decreasing values of valence. Experiment 2 revealed a congruency effect only for high arousal combinations and no interaction effects. This pattern was explained by a three parameter constant weight averaging model with greater weight for the auditory modality and a very low arousal value for the initial state. These results demonstrate key differences in audiovisual integration between valence and arousal.

KEYWORDS:

• Affect
• audiovisual integration
• congruency
• modality dominance
• negativity dominance

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes

1. Power analysis indicated that our sample size (n = 34) in each experiment has power of .98 to detect a moderate sized interaction effect (f = .25).

2. A parallel modality × valence two-way ANOVA was also conducted for the unimodal arousal ratings. The main effect of modality was significant, F(1, 33) = 28.20, p < .001,  = 0.46. The music stimuli had higher arousal than the video stimuli (M_music = 5.68, M_video  = 4.29). The main effect of valence was not significant, F(2, 66) = 3.30, p = .055, nor was the interaction, F(2, 66) = 1.02, p > .05. The valence conditions did not significantly differ on arousal (M_Positive  = 5.00, M_Neutral  = 4.83, M_Negative  = 5.13), although arousal for the neutral condition was somewhat lower than for the valence conditions.

3. Item-wise ANOVA was also conducted, and the pattern of significant results was the same as those of the subject-wise ANOVA reported here.

4. A parallel modality × arousal two-way ANOVA was conducted to examine if there are differences of valence ratings across conditions. Neither the main effect of modality, F(1, 33) = 0.22, p > .05, nor the main effect of arousal, F(2, 66) = 1.99, p > .05, were significant. A significant modality × arousal interaction was found, F(2, 66) = 9.64, p < .01,  = 0.226. Two separate ANOVAs were run for valence ratings of video arousal and music arousal. The main effect of video arousal was not significant, F(2, 66) = 3.13, p = .068, M_high  = 5.93, M_moderate  = 5.56, M_low  = 5.85. The main effect of music arousal was significant, F(2, 66) = 6.96, p < .01,  = 0.174, M_high  = 5.57, M_moderate  = 5.96, M_low  = 6.07. Although there was a significant interaction on valence ratings, these differences were small and taken into account in our modeling analyses.

5. Significant gender differences in emotional processing have been reported in the literature (e.g. Deng et al., 2016). Our study was not designed to examine gender differences, which may be an area for future research. For Experiment 1 (11 males, 23 females), we conducted separate ANOVAs by gender and found the same pattern of significant effects for males and females. This result suggests valence integration was the same for males and females. We did not conduct this analysis for Experiment 2, given that there were only 4 males.