Look who's talking? Sound changes gaze behaviour in a dynamic social scene

Abstract

Humans often look at other people in natural scenes, and previous research has shown that these looks follow the conversation and that they are sensitive to sound in audiovisual speech perception. In the present experiment, participants viewed video clips of four people involved in a discussion. By removing the sound, we asked whether auditory information would affect when speakers were fixated, how fixations between different observers were synchronized, and whether the eyes or mouth were looked at most often. The results showed that sound changed the timing of looks—by alerting observers to changes in conversation and attracting attention to the speaker. Clips with sound also led to greater attentional synchrony, with more observers fixating the same regions at the same time. However, looks towards the eyes of the people continued to dominate and were unaffected by removing the sound. These findings provide a rich example of multimodal social attention.

Keywords:

• Eye movements
• Social attention
• Scene perception
• Speech perception

This work was supported by a British Academy Small Grant to TF [grant number SG120844]. We also thank Ben Tatler and two anonymous reviewers for their excellent suggestions.

This work was supported by a British Academy Small Grant to TF [grant number SG120844]. We also thank Ben Tatler and two anonymous reviewers for their excellent suggestions.