Evaluating non-affective cross-modal congruence effects on emotion perception

ABSTRACT

Although numerous studies have shown that people are more likely to integrate consistent visual and auditory signals, the role of non-affective congruence in emotion perception is unclear. This registered report examined the influence of non-affective cross-modal congruence on emotion perception. In Experiment 1, non-affective congruence was manipulated by matching or mismatching gender between visual and auditory modalities. Participants were instructed to attend to emotion information from only one modality while ignoring the other modality. Experiment 2 tested the inverse effectiveness rule by including both noise and noiseless conditions. Across two experiments, we found the effects of task-irrelevant emotional signals from one modality on emotional perception in the other modality, reflected in affective congruence, facilitation, and affective incongruence effects. The effects were stronger for the attend-auditory compared to the attend-visual condition, supporting a visual dominance effect. The effects were stronger for the noise compared to the noiseless condition, consistent with the inverse effectiveness rule. We did not find evidence for the effects of non-affective congruence on audiovisual integration of emotion across two experiments, suggesting that audiovisual integration of emotion may not require automatic integration of non-affective congruence information.

KEYWORDS:

• Multisensory
• emotion
• affect
• audiovisual integration
• cross-modal congruence

Acknowledgements

We thank Christine E. Weber for helpful comments.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Open practices statements

The data and code for the present study are available at Open Science Framework (https://osf.io/2d4z9/).