Cognitive Walkthroughs: Understanding the Effect of Task-Description Detail on Evaluator Performance

Abstract

Inspection-based evaluation techniques are popular because they can be fast, require limited formal training, and can find numerous usability problems. Cognitive walkthroughs are one of the most studied techniques, and as a result, the technique has undergone a series of revisions. One such revision, made to speed the evaluation process and reduce the need for formal training in cognitive psychology, incorporated detailed step-by-step task descriptions to guide the evaluation process. This article reports on the first study that investigated the effects of this change when evaluators were learning to apply the technique. The results indicate that providing detailed task descriptions rather than shorter descriptions, as are often used in usability tests, significantly changes the types of problems found. Evaluators given detailed task descriptions found significantly more problems related to the feedback provided by the system but significantly fewer problems related to difficulties locating the necessary controls to complete a task. In addition, evaluators given detailed task descriptions found significantly more low-severity problems. Implications for both researchers and practitioners are discussed.