Dual Verbal Elicitation: The Complementary Use of Concurrent and Retrospective Reporting Within a Usability Test

Abstract

Verbal protocols are the primary tool for understanding users' task-solving behaviors during usability testing. A qualitative study that examined the utility of combining a concurrent and retrospective think-aloud within the same usability test is described. The results indicate that although there was significant overlap between the types of utterances produced during each think-aloud, the retrospective phase produced more verbalizations that were relevant to usability analysis, for example, helpful self-assessments of performance, yielding insights into the impact of encountered difficulties. However, a small number of less desirable utterance types emerged: hypothesising, rationalizing, and forgetting. When used together, both methods contributed to an understanding of usability issues; the concurrent phase yielded more usability issues overall, and the retrospective data improved the understanding of these by (a) reinforcement: users highlighted the impact of an issue on their experience, (b) elaboration: users would provide causal explanations of encountered difficulties, and (c) context: users provided information about the product's context of use.