Dual Verbal Elicitation: The Complementary Use of Concurrent and Retrospective Reporting Within a Usability Test

REFERENCES

• Als , B. S. , Jensen , J. J. and Skov , M. B. 2005 . “ Comparison of think-aloud and constructive interaction in usability testing with children ” . In Proceedings of the 2005 conference on Interaction design and children , 9 – 16 . New York , NY : ACM .
   Google Scholar
• Barnum , C. M. 2010 . Usability testing essentials: Ready, set … test! , Burlington , MA : Morgan Kaufmann .
   Google Scholar
• Boren , M. T. and Ramey , J. 2000 . Thinking aloud: Reconciling theory and practice . IEEE Transactions on Professional Communication , 43 : 261
   Web of Science ®Google Scholar
• Bowers , V. A. and Snyder , H. L. 1990 . “ Concurrent versus retrospective verbal protocols for comparing windows usability ” . In Proceedings of the Human Factor Society 34th Annual Meeting 1270 – 1274 .
   Google Scholar
• Chi , M. T. H. 1997 . Quantifying qualitative analyses of verbal data: A practical guide . The Journal of the Learning Sciences , 6 : 271 – 315 .
   Web of Science ®Google Scholar
• Chi , M. T. H. , de Leeuw , N. , Chiu , M. H. and LaVancher , C. 1994 . Eliciting self-explanations improves understanding . Cognitive Science , 18 : 439 – 477 .
   Web of Science ®Google Scholar
• Chilana , P. K. , Wobbrock , J. O. and Ko , A. J. 2010 . “ Understanding usability practices in complex domains ” . In Proceedings of the 28th International Conference on Human Factors in Computing Systems 2337 – 2346 .
   Google Scholar
• Chin , J. M. and Schooler , J. W. 2008 . Why do words hurt? Content process, and criterion shift accounts of verbal overshadowing . European Journal of Cognitive Psychology , 20 : 396 – 413 .
   Web of Science ®Google Scholar
• Cooke , L. 2010 . Assessing concurrent think-aloud protocol as a usability test method: A technical communication approach . IEEE Transactions on Professional Communication , 53 : 202 – 215 .
   Web of Science ®Google Scholar
• Cotton , D. and Gresty , K. 2006 . Reflecting on the think-aloud method for evaluating e-learning . British Journal of Educational Technology , 37 : 45 – 54 .
   Web of Science ®Google Scholar
• Eger , N. , Ball , L. J. , Stevens , R. and Dodd , J. 2007 . “ Cueing retrospective verbal reports in usability testing through eye-movement replay ” . In Proceedings of the 21st British HCI Group Annual Conference on People and Computers XXI: HCI … but Not as We Know It 129 – 137 .
   Google Scholar
• Elling , S. , Lentz , L. and de Jong , M. 2011 . “ Retrospective think-aloud method: using eye movements as an extra cue for participants' verbalizations ” . In Proceedings of the 2011 Annual Conference on Human Factors in Computing Systems , 1161 – 1170 . New York , NY : ACM .
   Google Scholar
• Ericsson , K. A. and Simon , H. A. 1984 . Protocol analysis: Verbal reports as data , Cambridge , MA : MIT Press .
   Google Scholar
• Ericsson , K. A. and Simon , H. A. 1993 . Protocol analysis: Verbal reports as data , Rev , Cambridge , MA : MIT Press .
   Google Scholar
• Følstad , A. and Hornbæk , K. 2010 . Work-domain knowledge in usability evaluation: Experiences with cooperative usability testing . Journal of Systems and Software , 83 : 2019 – 2030 .
   Web of Science ®Google Scholar
• Fox , M. C. , Ericsson , K. A. and Best , R. 2011 . Do procedures for verbal reporting of thinking have to be reactive? A meta-analysis and recommendations for best reporting methods . Psychological Bulletin , 137 : 316 – 344 .
   PubMed Web of Science ®Google Scholar
• Frøkjær , E. and Hornbæk , K. 2005 . “ Cooperative usability testing: Complementing usability tests with user-supported interpretation sessions ” . In CHI '05 Extended Abstracts on Human Factors in Computing Systems , 1383 – 1386 . New York , NY : ACM .
   Google Scholar
• Grossman , T. , Fitzmaurice , G. and Attar , R. 2009 . “ A survey of software learnability: metrics, methodologies and guidelines ” . In Proceedings of the 27th International Conference on Human Factors in Computing Systems , 649 – 658 . New York , NY : ACM .
   Google Scholar
• Guan , Z. , Lee , S. , Cuddihy , E. and Ramey , J. 2006 . “ The validity of the stimulated retrospective think-aloud method as measured by eye tracking ” . In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems , 1253 – 1262 . New York , NY : ACM .
   Google Scholar
• Gulliksen , J. , Boivie , I. and Göransson , B. 2006 . Usability professionals-current practices and future development . Interacting with Computers , 18 : 568 – 600 .
   Web of Science ®Google Scholar
• Hertzum , M. 2006 . Problem prioritization in usability evaluation: From severity assessments toward impact on design . International Journal of Human–Computer Interaction , 21 : 125 – 146 .
   Web of Science ®Google Scholar
• Hertzum , M. , Hansen , K. D. and Andersen , H. H. K. 2009 . Scrutinising usability evaluation: Does thinking aloud affect behaviour and mental workload? . Behaviour and Information Technology , 28 : 165 – 181 .
   Web of Science ®Google Scholar
• Hertzum , M. and Jacobsen , N. E. 2001 . The evaluator effect: a chilling fact about usability evaluation methods . International Journal of Human-Computer Interaction , 13 : 421 – 443 .
   Web of Science ®Google Scholar
• Karahasanović , A. , Hinkel , U. N. , Sjøberg , D. I. K. and Thomas , R. 2009 . Comparing of feedback-collection and think-aloud methods in program comprehension studies . Behaviour and Information Technology , 28 : 139 – 164 .
   Web of Science ®Google Scholar
• Kumar , J. , Yammiyavar , P. and Nielsen , J. 2008 . Mind Tape technique—A usability evaluation method for tracing cognitive processes in cross cultural settings . e-Minds , 1 : 69 – 85 .
   Google Scholar
• Kuusela , H. and Paul , P. 2000 . A comparison of concurrent and retrospective verbal protocol analysis . The American Journal of Psychology , 113 : 387 – 404 .
   PubMed Web of Science ®Google Scholar
• Leow , R. P. 2002 . Models, attention, and awareness in Sla . Studies in Second Language Acquisition , 24 : 113 – 119 .
   Google Scholar
• Maguire , M. 2001 . Context of use within usability activity . International Journal of Human–Computer Studies , 55 : 453 – 483 .
   Web of Science ®Google Scholar
• Makri , S. , Blandford , A. and Cox , A. L. 2011 . This is what I'm doing and why: Methodological reflections on a naturalistic think-aloud study of interactive information behaviour . Information Processing & Management , 47 : 336 – 348 .
   Web of Science ®Google Scholar
• McDonald , S. , Edwards , H. M. and Zhao , T. 2012 . Exploring think-alouds in usability testing: An international survey . IEEE Transactions of Professional Communication , 55 : 1 – 17 .
   Google Scholar
• McDonald , S. , Monahan , K. and Cockton , G. 2006 . “ Rapid contextual evaluation ” . In Proceedings of the 4th Nordic Conference on Human-Computer Interaction 437 – 440 .
   Google Scholar
• Nørgaard , M. and Hornbæk , K. 2006 . “ What do usability evaluators do in practice?: An explorative study of think-aloud testing ” . In Proceedings of the 6th conference on Designing Interactive Systems 209 – 218 .
   Google Scholar
• Nørgaard, M., & Hornbæk, K. (2008). Working together to improve usability: Challenges and best practices. Copenhagen University Technical Report. University of Copenhagen, Denmark. http://www.diku.dk/OLD/publikationer/tekniske.rapporter/rapporter/08-03.pdf (http://www.diku.dk/OLD/publikationer/tekniske.rapporter/rapporter/08-03.pdf)
   Google Scholar
• Norman , D. 2002 . The design of everyday things , New York , NY : Basic Books .
   Google Scholar
• Olmsted-Hawala , E. L. , Murphy , E. D , Hawala , S. and Ashenfelter , K. T. 2010 . “ Think-aloud protocols: a comparison of three think-aloud protocols for use in testing data-dissemination web sites for usability ” . In Proceedings of the 28th international Conference on Human Factors in Computing Systems 2381 – 2390 .
   Google Scholar
• Page , C. and Rahimi , M. 1995 . Concurrent and retrospective verbal protocols in usability testing: is there value added in collecting both . Human Factors and Ergonomics Society Annual Meeting Proceedings , 39 : 223 – 227 .
   Google Scholar
• Peute , L. W. , de Keizer , N. F. and Jaspers , M. W. M. 2010 . Cognitive evaluation of a physician data query tool for a national ICU registry: Comparing two think aloud variants and their application in redesign . Studies in Health Technology and Informatics , 160 : 309 – 313 .
   PubMedGoogle Scholar
• Russo , J. E. , Johnson , E. J. and Stephens , D. L. 1989 . The validity of verbal protocols . Memory and Cognition , 17 : 759 – 769 .
   PubMed Web of Science ®Google Scholar
• Schooler , J. W. 2011 . Introspecting in the spirit of William James: Comment on Fox, Ericsson, and Best (2011) . Psychological Bulletin , 137 : 345 – 350 .
   PubMed Web of Science ®Google Scholar
• Shi , Q. 2008 . “ A field study of the relationship and communication between Chinese evaluators and users in thinking aloud usability tests ” . In Proceedings of the 5th Nordic Conference on Human–Computer Interaction: Building Bridges 344 – 352 .
   Google Scholar
• Shrimpton-Smith , T. , Zaman , B. and Geerts , D. 2008 . Coupling the users: The benefits of paired user testing for IDV . International Journal of Human–Computer Interaction , 24 : 197 – 213 .
   Web of Science ®Google Scholar
• Tamler , H. 1998 . How (much) to intervene in a usability testing session . Common Ground , 8 : 11 – 15 .
   Google Scholar
• Taylor , K. L. and Dionne , J. P. 2000 . Accessing problem-solving strategy knowledge: The complementary use of concurrent verbal protocols and retrospective debriefing . Journal of Educational Psychology , 92 : 413 – 425 .
   Web of Science ®Google Scholar
• van den Haak , M. J. 2008 . A penny for your thoughts: Investigating the validity and reliability of think-aloud protocols for usability testing (Unpublished doctoral thesis) , Enschede , , the Netherlands : University of Twente .
   Google Scholar
• van den Haak , M. J. , de Jong , M. and Schellens , P. J. 2003 . Retrospective vs. concurrent think-aloud protocols: Testing the usability of an online library catalogue . Behaviour and Information Technology , 22 : 339 – 351 .
   Web of Science ®Google Scholar
• van den Haak , M. J. , de Jong , M. D. T. and Schellens , P. J. 2006 . Constructive Interaction: An analysis of verbal interaction in a usability setting . IEEE Transactions on Professional Communication , 49 : 311 – 324 .
   Google Scholar
• van den Haak , M. J. , de Jong , M. D.T. and Schellens , P. J. 2009 . Evaluating municipal websites: A methodological comparison of three think-aloud variants . Government Information Quarterly , 26 : 193 – 202 .
   Web of Science ®Google Scholar
• Wilson , T. D. 1994 . The proper protocol: Validity and completeness of verbal reports . Psychological Science , 5 : 249 – 252 .
   Web of Science ®Google Scholar
• Yang , S. C. 2003 . Reconceptualizing think-aloud methodology: Refining the encoding and categorizing techniques via contextualized perspectives . Computers in Human Behaviour , 19 : 95 – 115 .
   Web of Science ®Google Scholar
• Yinger , R. J. 1986 . Examining thought in action: A theoretical and methodological critique of research on interactive teaching . Teaching and Teacher Education , 2 : 263 – 282 .
   Google Scholar
• Zhao , T. and McDonald , S. 2010 . “ Keep talking: An analysis of participant utterances gathered using two concurrent think-aloud methods ” . In Proceedings of the 6th Nordic Conference on Human-Computer Interaction: Extending Boundaries 581 – 590 .
   Google Scholar
• Zhao , T. , McDonald , S. and Edwards , H. M. 2012 . The impact of two different think-aloud instructions in a usability test: A case of just following orders? . Behaviour and Information Technology. Advance online publication. , doi: 10.1080/0144929X.2012.708786
   Google Scholar