Prosody in speech as a source of referential information

ABSTRACT

Although prosody has primarily been assumed to convey information regarding linguistic structure and speakers’ emotional state, increasing evidence suggests that prosody also conveys referential details. We examined the extent to which language users produce and infer information from prosodic correlates to perceptual details in the visual modality, specifically colour brightness. In Experiments 1 and 2, speakers labelled colours that varied in brightness with either familiar colour names (e.g. red; Exp. 1) or novel words (e.g. blicket; Exp. 2). Speakers used higher pitch, shorter duration, and higher amplitude for novel words, but not for familiar colour names, when labelling brighter versus darker shades. Listeners in Experiment 3 reliably inferred the intended target colour referent from the recorded utterances obtained in Experiment 2. Findings suggest that prosody reflects cross-modal correspondences between auditory and visual domains and, like a type of vocal gesture, provides an additional channel of information that resolves referential ambiguity.

KEYWORDS:

• Prosody
• semantics
• spoken language
• cross-modal correspondences
• linguistic reference

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes

1. Participants were screened for hearing and speech disorders via self-reported responses on a questionnaire that was completed prior to the start of the experiment. Participants were not screened for colour blindness. However, before the start of Experiments 1 and 2, participants completed a practice trial during which they were told to label the darkest, intermediate, or brightest shade of a grey colour spectrum with either an English colour label (Experiment 1) or novel word (Experiment 2). The experimenter also explained that all presented spectra would include shades that progressively increased in brightness from left to right. In Experiment 3, participants were told that they would be choosing between one bright swatch and one dark swatch on each trial. These steps ensured that participants were aware of differences in brightness among shades regardless of the specific colour instantiated.

2. Levels of blue in the sixth and seventh shades of the yellow spectrum (RGB: 225, 255, 80 and 255, 255, 112) and the fifth and ninth shades of the purple spectrum (127, 0, 225 and 229. 204, 225) were manually adjusted to reduce perceived similarity between consecutive shades.

3. All utterances were amplitude normalised using the same procedure both within and across speakers in order to account for slight shifts in distance to the microphone over the course of an experimental session. Such shifts primarily affect overall amplitude but not pitch and duration (Zollinger, Podos, Nemeth, Goller, & Brumm, 2012). Amplitude normalisation did not affect pitch and duration measurements and preserved relative amplitude differences within and across utterances, ensuring that any differences across conditions were due to differences in the vocal effort (speaking more loudly or softly) and not due to distance from the microphone.

4. Mean F0 measures were obtained in PRAAT using an autocorrelation pitch extraction algorithm (Boersma, 1993), with minimum and maximum pitch settings set at 75 and 500 Hz, respectively. Minimum and maximum amplitude settings were set at 50 and 100 dB, respectively.

5. Stimuli in Experiment 3 were presented at 75 dB SPL to ensure clear auditory presentation and to preserve a comfortable listening level.