Retrospective protocols in usability testing: a comparison of Post-session RTA versus Post-task RTA reports

References

• Block, R. A., P. A. Hancock, and D. Zakay. 2010. “How Cognitive Load Affects Duration Judgments: A Meta-analytic Review.” Acta Psychologica 134 (3): 330–343. doi: 10.1016/j.actpsy.2010.03.006
   PubMed Web of Science ®Google Scholar
• Boren, M. T., and J. Ramey. 2000. “Thinking Aloud: Reconciling Theory and Practice.” IEEE Transactions on Professional Communication 43 (3): 261–278. doi: 10.1109/47.867942
   Web of Science ®Google Scholar
• Bowers, V. A., and H. L. Snyder. 1990. “Concurrent versus Retrospective Verbal Protocols for Comparing Windows Usability.” In Proceedings of the Human Factors Society 34th Annual Meeting, 1270–1274 Santa Monica, CA: HFES.
   Google Scholar
• Chilana, P. K., J. O. Wobbrock, and A. J. Ko. 2010. “Understanding Usability Practices in Complex Domains.” In Proceedings of the 28th International Conference on Human Factors in Computing Systems, CHI’10, Atlanta, GA, USA, April 10–15, 2337–2346. New York: ACM.
   Google Scholar
• Cotton, D., and K. Gresty. 2006. “Reflecting on the Think-aloud Method for Evaluating E-learning.” British Journal of Educational Technology 37 (1): 45–54. doi: 10.1111/j.1467-8535.2005.00521.x
   Web of Science ®Google Scholar
• Eger, N., L. J. Ball, R. Stevens, and J. Dodd. 2007. “Cueing Retrospective Verbal Reports in Usability Testing through Eye-movement Replay.” In Proceedings of the 21st British HCI Group Annual Conference on People and Computers XXI: HCI … but Not as We Know It, HCI’07, 129–137. Swindon: British Computer Society.
   Google Scholar
• Elling, S., L. Lentz, and M. de Jong. 2011. “Retrospective Think-aloud Method: Using Eye Movements as an Extra Cue for Participants’ Verbalisations.” In Proceedings of the 2011 Annual Conference on Human Factors in Computing Systems, CHI’11, Vancouver, BC, Canada, May 7–12, 1161–1170. New York: ACM.
   Google Scholar
• Ericsson, K. A., and H. A. Simon. 1984. Protocol Analysis: Verbal Reports as Data. Cambridge: MIT Press.
   Google Scholar
• Ericsson, K. A., and H. A. Simon. 1993. Protocol Analysis: Verbal Reports as Data. Revised ed. Cambridge: MIT Press.
   Google Scholar
• Fox, M. C., K. A. Ericsson, and R. Best. 2011. “Do Procedures for Verbal Reporting of Thinking Have to be Reactive? A Meta-analysis and Recommendations for Best Reporting Methods.” Psychological Bulletin 137 (2): 316–344. doi: 10.1037/a0021663
   PubMed Web of Science ®Google Scholar
• Følstad, A., and K. Hornbæk. 2010. “Work-domain Knowledge in Usability Evaluation: Experiences with Cooperative Usability Testing.” Journal of Systems and Software 83 (11): 2019–2030. doi: 10.1016/j.jss.2010.02.026
   Web of Science ®Google Scholar
• Frøkjær, E., and K. Hornbæk. 2005. “Cooperative Usability Testing: Complementing Usability Tests with User-supported Interpretation Sessions.” In CHI’05 Extended Abstracts on Human Factors in Computing Systems, CHI EA’05, Portland, Oregon, USA, April 2–7, 1383–1386. New York: ACM.
   Google Scholar
• Gerjets, P., Y. Kammerer, and B. Werner. 2011. “Measuring Spontaneous and Instructed Evaluation Processes During Web Search: Integrating Concurrent Think-Aloud Protocols and eye-Tracking Data.” Learning and Instruction 21 (2): 220–231. doi: 10.1016/j.learninstruc.2010.02.005
   Web of Science ®Google Scholar
• Gray, W. D., and M. C. Salzman. 1998. “Damaged Merchandise? A Review of Experiments that Compare Usability Evaluation Methods.” Human-Computer Interaction 13 (3): 203–261. doi: 10.1207/s15327051hci1303_2
   Web of Science ®Google Scholar
• Guan, Z., S. Lee, E. Cuddihy, and J. Ramey. 2006. “The Validity of the Stimulated Retrospective Think-aloud Method as Measured by Eye Tracking.” In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 1253–1262. New York, NY: ACM.
   Google Scholar
• van den Haak, M. J., M. de Jong, and P. J. Schellens. 2003. “Retrospective vs. Concurrent Think-aloud Protocols: Testing the Usability of an Online Library Catalogue.” Behaviour & Information Technology 22 (5): 339–351. doi: 10.1080/0044929031000
   Web of Science ®Google Scholar
• van den Haak, M. J., M. D. T. de Jong, and P. J. Schellens. 2004. “Employing Think-aloud Protocols and Constructive Interaction to Test the Usability of Online Library Catalogues: A Methodological Comparison.” Interacting with Computers 16 (6): 1153–1170. doi: 10.1016/j.intcom.2004.07.007
   Web of Science ®Google Scholar
• van den Haak, M. J., M. D. de Jong, and P. J. Schellens. 2007. “Evaluation of an Informational Web Site: Three Variants of the Think-aloud Method Compared.” Technical Communication 54 (1): 58–71.
   Web of Science ®Google Scholar
• van den Haak, M. J., M. D. T. de Jong, and P. J. Schellens. 2009. “Evaluating Municipal Websites: A Methodological Comparison of Three Think-aloud Variants.” Government Information Quarterly 26 (1): 193–202. doi: 10.1016/j.giq.2007.11.003
   Web of Science ®Google Scholar
• Hertzum, M., P. Borlund, and K. B. Kristoffersen. 2015. What Do Thinking-aloud Participants Say? A Comparison of Moderated and Unmoderated Usability Sessions. International Journal of Human-Computer Interaction 31 (9): 557–570. doi:10.1080/10447318.2015.1065691.
   Web of Science ®Google Scholar
• Hertzum, M., K. D. Hansen, and H. H. K. Andersen. 2009. “Scrutinising Usability Evaluation: Does Thinking Aloud Affect Behaviour and Mental Workload?” Behaviour & Information Technology 28 (2): 165–181. doi: 10.1080/01449290701773842
   Web of Science ®Google Scholar
• Hertzum, M., and K. D. Holmegaard. 2013. “Thinking Aloud in the Presence of Interruptions and Time Constraints.” International Journal of Human-Computer Interaction 29: 351–364. doi: 10.1080/10447318.2012.711705
   Web of Science ®Google Scholar
• Leow, R. P. 2002. “Models, Attention and Awareness in Sla.” Studies in Second Language Acquisition 24 (1): 113–119. doi: 10.1017/S0272263102001055
   Google Scholar
• McDonald, S., H. Edwards, and T. Zhao. 2012. “Exploring Think-alouds in Usability Testing: An International Survey.” IEEE Transactions on Professional Communication 55 (1): 2–19. doi: 10.1109/TPC.2011.2182569
   Web of Science ®Google Scholar
• McDonald, S., and H. Petrie. 2013. “The Effect of Global Instructions on Think-aloud Testing.” In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, Paris, France, April 27–May 2, 2941–2944. New York, NY: ACM.
   Google Scholar
• McDonald, S., T. Zhao, and H. M. Edwards. 2013. Dual Verbal Elicitation: The Complementary use of Concurrent and Retrospective Reporting within a Usability Test. International Journal of Human Computer Interaction, 29: 647–660. doi: 10.1080/10447318.2012.758529
   Web of Science ®Google Scholar
• McDonald, S., T. Zhao, and H. M. Edwards. 2015. “Look Who’s Talking: Evaluating the Utility of Interventions During an Interactive Think-aloud.” Interacting with Computers 28: 387–403. doi:10.1093/iwc/iwv014.
   Web of Science ®Google Scholar
• Nørgaard, M., and K. Hornbæk. 2006. “What Do Usability Evaluators Do in Practice? An Explorative Study of Think-aloud Testing.” In Proceedings of the 6th Conference on Designing Interactive Systems, University Park, Pennsylvania, June 26–28, 209–218. New York: ACM Press.
   Google Scholar
• Ohnemus, K. R., and D. W. Biers. 1993. “Retrospective Versus Concurrent Thinking-out-Loud in Usability Testing.” Proceedings of Human Factors and Ergonomics Society Annual Meeting 37 (17): 1127–1131.
   Google Scholar
• Olmsted-Hawala, E. L., E. D. Murphy, S. Hawala, and K. T. Ashenfelter. 2010. “Think-aloud Protocols: A Comparison of Three Think-aloud Protocols for use in Testing Data-dissemination Web Sites for Usability.” In Proceedings of the 28th International Conference on Human Factors in Computing Systems, 2381–2390 Atlanta: ACM Press.
   Google Scholar
• Page, C., and M. Rahimi. 1995. “Concurrent and Retrospective Verbal Protocols in Usability Testing: Is there Value Added in Collecting Both?” Proceedings of Human Factors and Ergonomics Society Annual Meeting 39 (4): 223–227. doi: 10.1177/154193129503900401
   Google Scholar
• Shi, Q. 2008. “A Field Study of the Relationship and Communication between Chinese Evaluators and Users in Thinking Aloud Usability Tests.” In Proceedings of the 5th Nordic Conference on Human–computer Interaction: Building Bridges, Lund, Sweden, October 20–22, 344–352. New York: ACM Press.
   Google Scholar
• Taylor, K. L., and J. P. Dionne. 2000. “Accessing Problem-solving Strategy Knowledge: The Complementary use of Concurrent Verbal Protocols and Retrospective Debriefing.” Journal of Educational Psychology 92 (3): 413–425. doi: 10.1037/0022-0663.92.3.413
   Web of Science ®Google Scholar
• Yang, S. C. 2003. “Reconceptualising Think-aloud Methodology: Refining the Encoding and Categorising Techniques via Contextualised Perspectives.” Computers in Human Behaviour 19 (1): 95–115. doi: 10.1016/S0747-5632(02)00011-0
   Web of Science ®Google Scholar
• Zhao, T., and S. McDonald. 2010. “Keep Talking: An Analysis of Participant Utterances Gathered using Two Concurrent Think-aloud Methods.” In Proceedings of the 6th Nordic Conference on Human-Computer Interaction: Extending Boundaries, NordicCHI’10, Reykjavik, Iceland, October 16–20, 581–590. New York: ACM.
   Google Scholar
• Zhao, T., S. McDonald, and H. M. Edwards. 2014. “The Impact of Two Different Think-aloud Instructions in a Usability Test: A Case of Just Following Orders?” Behaviour and Information Technology 33 (2): 163–183. doi:10.1080/0144929X.2012.708786.
   Web of Science ®Google Scholar