Visual and acoustic information supporting a happily expressed speech-in-noise advantage

ABSTRACT

There was an advantage found for recognizing happily expressed audio and audio-visual speech-in-noise relative to speech spoken with neutral or sad expressions. The advantage of happily expressed speech was explored in a set of visual and acoustic manipulations designed to isolate the potential contributions from each signal. For this research, a replication of previous research with the happily expressed speech advantage was completed with a novel inclusion of a learning paradigm. Additional experiments directly investigated the role of the fundamental frequency of the voice for affect, and the affective facial components with the eyes and mouth. We found that the happily expressed speech advantage persisted despite constraints to the amount of frequency modulation and with distortions to the talker's face. These findings seem largely attributable to the influences of affect on low-level acoustical and articulatory information, with only a very subtle role of approach/withdrawal motivation.

KEYWORDS:

• Speech
• Audio-visual
• Perception
• Affect
• Multimodal
• Learning