Scrutinising usability evaluation: does thinking aloud affect behaviour and mental workload?

References

• Berry , D. C. and Broadbent , D. E. 1990 . The role of instruction and verbalisation in improving performance on complex search tasks . Behaviour & Information Technology , 9 ( 3 ) : 175 – 190 .
   Web of Science ®Google Scholar
• Boren , T. and Ramey , J. 2000 . Thinking aloud: reconciling theory and practice . IEEE Transactions on Professional Communication , 43 ( 3 ) : 261 – 278 .
   Web of Science ®Google Scholar
• Campbell , D. J. 1988 . Task complexity: a review and analysis . Academy of Management Review , 13 ( 1 ) : 40 – 52 .
   Web of Science ®Google Scholar
• Chi , M. T.H. , Bassok , M. , Lewis , M. , Reimann , P. and Glaser , R. 1989 . Self-explanations: how students study and use examples in learning to solve problems . Cognitive Science , 13 ( 2 ) : 145 – 182 .
   Web of Science ®Google Scholar
• Chi , M. T.H. , De Leeuw , N. , Chiu , M.-H. and LaVancher , C. 1994 . Eliciting self-explanations improves understanding . Cognitive Science , 18 ( 3 ) : 439 – 477 .
   Web of Science ®Google Scholar
• Cockton , G. and Woolrych , A. 2002 . Sale must end: should discount methods be cleared off HCI's shelves? . ACM Interactions , 9 ( 5 ) : 13 – 18 .
   Google Scholar
• DeShon , R. P. , Chan , D. and Weissbein , D. A. 1995 . Verbal overshadowing effects on Raven's advanced progressive matrices: evidence for multidimensional performance determinants . Intelligence , 21 ( 2 ) : 135 – 155 .
   Web of Science ®Google Scholar
• Dumas , J. S. 2003 . “ User-based evaluations ” . In The human-computer interaction handbook , Edited by: Jacko , J. and Sears , A. 1093 – 1117 . Mahwah, NJ : Erlbaum .
   Google Scholar
• Dumas , J. S. and Redish , J. C. 1999 . A practical guide to usability testing , Revised edition , Exeter, , UK : Intellect .
   Google Scholar
• Ericsson , K. A. and Simon , H. A. 1980 . Verbal reports as data . Psychological Review , 87 ( 3 ) : 215 – 251 .
   Web of Science ®Google Scholar
• Ericsson , K. A. and Simon , H. A. 1993 . Protocol analysis: verbal reports as data , Revised edition , Cambridge, MA : MIT Press .
   Google Scholar
• Fr⊘kjær , E. and Hornbæk , K. . Cooperative usability testing: complementing usability tests with user-supported interpretation sessions . Extended abstracts of the CHI 2005 conference on human factors in computing systems . April 2–5 , Portland, Oregon. pp. 1383 – 1386 . New York : ACM Press .
   Google Scholar
• Goldberg , J. H. and Kotval , X. P. 1999 . Computer interface evaluation using eye movements: methods and constructs . International Journal of Industrial Ergonomics , 24 ( 6 ) : 631 – 645 .
   Web of Science ®Google Scholar
• Guan , Z. , Lee , S. , Cuddihy , E. and Ramey , J. . The validity of the stimulated retrospective think-aloud method as measured by eye tracking . Proceedings of the CHI 2006 conference on human factors in computing systems . April 22–27 , Montreal, CA. pp. 1253 – 1262 . New York : ACM Press .
   Google Scholar
• Hart , S. G. and Staveland , L. E. 1988 . “ Development of NASA-TLX (task load index): results of empirical and theoretical research ” . In Human mental workload , Edited by: Hancock , P. A. and Meshkati , N. 139 – 183 . Amsterdam : Elsevier .
   Google Scholar
• Held , J. E. and Biers , D. W. . Software usability testing: do evaluator intervention and task structure make any difference? . Proceedings of the human factors society 36th annual meeting . October 12–16 , Atlanta, GA. pp. 1215 – 1219 . Santa Monica, CA : HFS .
   Google Scholar
• Hendy , K. C. , Hamilton , K. M. and Landry , L. N. 1993 . Measuring subjective workload: when is one scale better than many? . Human Factors , 35 ( 4 ) : 579 – 601 .
   Web of Science ®Google Scholar
• James , W. 1890 . The principles of psychology , New York : Holt .
   Google Scholar
• Just , M. A. and Carpenter , P. A. 1980 . A theory of reading: from eye fixations to comprehension . Psychological Review , 87 ( 4 ) : 329 – 354 .
   PubMed Web of Science ®Google Scholar
• Krahmer , E. and Ummelen , N. 2004 . Thinking about thinking aloud: a comparison of two verbal protocols for usability testing . IEEE Transactions on Professional Communication , 47 ( 2 ) : 105 – 117 .
   Google Scholar
• Landis , J. R. and Koch , G. G. 1977 . The measurement of observer agreement for categorical data . Biometrics , 33 ( 1 ) : 159 – 174 .
   PubMed Web of Science ®Google Scholar
• Lesaigle , E. M. and Biers , D. W. . Effect of type of information on real time usability evaluation: implications for remote usability testing . Proceedings of the XIVth triennial congress of the international ergonomics association and 44th annual meeting of the human factors and ergonomics society . 29 July–4 August , San Diego, CA. pp. 6-585 – 6-588 . Santa Monica, CA : HFES .
   Google Scholar
• Lewis , C. 1982 . Using the ‘thinking-aloud’ method in cognitive interface design. Research report RC9265 , Yorktown Heights, NY : IBM Watson Research Center .
   Google Scholar
• Meissner , C. A. and Brigham , J. C. 2001 . A meta-analysis of the verbal overshadowing effect in face identification . Applied Cognitive Psychology , 15 ( 6 ) : 603 – 616 .
   Web of Science ®Google Scholar
• Monk , A. , Wright , P. , Haber , J. and Davenport , L. 1993 . Improving your human-computer interface: a practical technique , New York : Prentice Hall .
   Google Scholar
• Nielsen , J. 1993 . Usability engineering , Boston : Academic Press .
   Google Scholar
• Nisbett , R. E. and Wilson , T. D. 1977 . Telling more than we can know: verbal reports on mental processes . Psychological Review , 84 ( 3 ) : 231 – 259 .
   Web of Science ®Google Scholar
• Nygren , T. E. 1991 . Psychometric properties of subjective workload measurement techniques: implications for their use in the assessment of perceived mental workload . Human Factors , 33 ( 1 ) : 17 – 33 .
   Web of Science ®Google Scholar
• N⊘rgaard , M. and Hornbæk , K. . What do usability evaluators do in practice? An explorative study of think-aloud testing . Proceedings of the sixth DIS conference on designing interactive systems . June 26–28 , University Park, PA. pp. 209 – 218 . New York : ACM Press .
   Google Scholar
• Rayner , K. 1998 . Eye movements in reading and information processing: 20 years of research . Psychological Bulletin , 124 ( 3 ) : 372 – 422 .
   PubMed Web of Science ®Google Scholar
• Rhenius , D. and Deffner , G. . Evaluation of concurrent thinking aloud using eye-tracking data . Proceedings of the human factors society 34th annual meeting . October 8–12 , Orlando, FL. pp. 1265 – 1269 . Santa Monica, CA : HFS .
   Google Scholar
• Russo , J. E. , Johnson , E. J. and Stephens , D. L. 1989 . The validity of verbal protocols . Memory and Cognition , 17 ( 6 ) : 759 – 769 .
   PubMed Web of Science ®Google Scholar
• Salvucci , D. D. and Goldberg , J. H. . Identifying fixations and saccades in eye-tracking protocols . Proceedings of the eye tracking research and applications symposium 2000 . November 6–8 , Palm Beach Gardens, FL. pp. 71 – 78 . New York : ACM Press .
   Google Scholar
• Schooler , J. W. 2002 . Verbalization produces a transfer inappropriate processing shift . Applied Cognitive Psychology , 16 ( 8 ) : 989 – 997 .
   Web of Science ®Google Scholar
• Schooler , J. W. and Engstler-Schooler , T. Y. 1990 . Verbal overshadowing of visual memories: some things are better left unsaid . Cognitive Psychology , 22 ( 1 ) : 36 – 71 .
   PubMed Web of Science ®Google Scholar
• Spool , J. , Scanlon , T. , Schroeder , W. , Snyder , C. and DeAngelo , T. 1999 . Web site usability: a designer's guide , San Francisco, CA : Morgan Kaufmann .
   Google Scholar
• Van den Haak , M. J. , de Jong , M. D.T. and Schellens , P. J. 2003 . Retrospective vs. concurrent think-aloud protocols: testing the usability of an online library catalogue . Behaviour & Information Technology , 22 ( 5 ) : 339 – 351 .
   Web of Science ®Google Scholar
• Van den Haak , M. J. , de Jong , M. D.T. and Schellens , P. J. 2004 . Employing think-aloud protocols and constructive interaction to test the usability of online library catalogues: a methodological comparison . Interacting with Computers , 16 ( 6 ) : 1153 – 1170 .
   Web of Science ®Google Scholar
• Wilson , T. D. and Schooler , J. W. 1991 . Thinking too much: introspection can reduce the quality of preferences and decisions . Journal of Personality and Social Psychology , 60 ( 2 ) : 181 – 192 .
   PubMed Web of Science ®Google Scholar
• Wixon , D. 2003 . Evaluating usability methods: why the current literature fails the practitioner . ACM Interactions , 10 ( 4 ) : 29 – 34 .
   Google Scholar
• Wood , R. E. 1986 . Task complexity: definition of the construct . Organizational Behavior and Human Decision Processes , 37 ( 1 ) : 60 – 82 .
   Web of Science ®Google Scholar
• Wright , R. B. and Converse , S. A. . Method bias and concurrent verbal protocol in software usability testing . Proceedings of the human factors society 36th annual meeting . October 12–16 , Atlanta, GA. pp. 1220 – 1224 . Santa Monica, CA : HFS .
   Google Scholar