Thinking Aloud in the Presence of Interruptions and Time Constraints

Abstract

Thinking aloud is widely used for usability evaluation, and its reactivity is therefore important to the quality of evaluation results. This study investigates whether thinking aloud (i.e., verbalization at Levels 1 and 2) affects the behavior of users who perform tasks that involve interruptions and time constraints, two frequent elements of real-world activities. The study finds that the presence of auditory, visual, audiovisual, or no interruptions interacts with thinking aloud for task solution rate, task completion time, and participants' fixation rate. Thinking-aloud participants also spend longer responding to interruptions than control participants. Conversely, the absence or presence of time constraints does not interact with thinking aloud, suggesting that time pressure is less likely to make thinking aloud reactive than previously assumed. These results inform practitioners faced with the decision to either restrict verbalizations in usability evaluation to thinking aloud to avoid reactivity or relax the constraints on verbalization to obtain additional information.

ACKNOWLEDGMENTS

The first author provided the initial idea for the study, analyzed the data, and wrote most of the article. The second author planned, set up, and ran the experimental sessions. The authors took equal part in refining the initial idea for the study, and both authors critically read and revised draft versions of the article. We are grateful to Alexandre Alapetite and Steen Weber, who helped solve several issues about the use of the eye-tracking equipment, and to Signe Arnklit, who recruited the participants for the experiment. Special thanks are due to the participants.