Using Eye Movements to Study the Reading of Subtitles in Video

ABSTRACT

This article reports the first eye-movement experiment to examine how the presence versus absence of concurrent video content and presentation speed affect the reading of subtitles. Results indicated that participants adapted their visual routines to examine video content while simultaneously prioritizing the reading of subtitles, especially when the latter was displayed only briefly. Although decisions about when and where to move the eyes largely remained under local (cognitive) control, this control was also modulated by global task demands, suggesting an integration of local and global eye-movement control. The theoretical and pedagogical implications of these findings are discussed, and we also briefly describe a new theoretical framework for understanding all forms of multimodal reading, including the reading of subtitles in video.

Ethics statement

The present study was approved by the ethics committee of Macquarie University in accordance with National Statement on Ethical Conduct in Human Research (2007). The experiment and all planned analyses reported in this article have been preregistered (https://osf.io/9v8cy). Preliminary results of analyses reported in this article were presented at the SSSR conference in 2019. The authors declared that the findings reported in the manuscript are original and have not been published previously, and that the manuscript is not being simultaneously submitted elsewhere.

Informed Consent

All participants provided informed consent prior to their participation.

Conflict of Interest

The authors declared that there is no conflict of interest with respect to the authorship or the publication of this article.

Notes

1. This convention is referred to as the “six-second rule” because it allows two lines of 37 characters to remain visible on the screen for 6 seconds.

2. Subtitling guidelines recommend that subtitles should not be displayed on the screen across scene changes because this may cause the viewer to re-read the subtitle (Díaz Cintas & Remael, 2007; Robson, 2004).

3. Here are example models using a maximized random-effect structure for: (1) global analyses: DV ~ Video Presence * Subtitle Speed + (Video Presence * Subtitle Speed | Participants) + (Video Presence * Subtitle Speed | Subtitles); (2) frequency and length analyses: DV ~ Video Presence * Subtitle Speed * Frequency * Length + (Video Presence * Subtitle Speed * Frequency * Length | Participants) + (Video Presence * Subtitle Speed | Words); and (3) wrap-up effect analyses: DV ~ Video Presence * Subtitle Speed * Word Position + (Video Presence * Subtitle Speed * Word Position | Participants) + (Video Presence * Subtitle Speed | Words). Because the six video clips were all selected from the Planet Earth series and thus comparable, a single random effect variable was coded for each combination of video and subtitles in our global analyses (assuming no inherent video-subtitle hierarchy), and for each combination of video, subtitles, and words in our word-based analyses.

4. The mean across-condition accuracies of these six participants were all above 0.4, suggesting that the by-condition accuracy of less than 0.4 was due to our video-presence and subtitle-speed manipulations. However, to ensure that our eye-movement data obtained was representative of equally engaged reading and comprehension across our conditions, these participants’ data were removed from our eye-movement analyses.

5. Wrap-up effects are typically examined by comparing eye-movement measures on specific target words from two positions within a sentence or clause: their middle vs. end. This method allows one to control other properties of the target words (e.g., their frequency). However, because our study used real video subtitles, this method of calculating wrap-up effects was not possible and so we instead simply compared our two eye-movement measures on words from the middles versus endings of the same subtitles, with their average zip frequencies being 5.4 versus 4.6, respectively, and their average lengths being 4.9 versus 5.9 letters, respectively.

Additional information

Funding

This study was supported by a grant from Australian Research Council [DP190100719] awarded to the third author. The first author was supported by a PhD scholarship from Macquarie University.