What Do Thinking-Aloud Participants Say? A Comparison of Moderated and Unmoderated Usability Sessions

REFERENCES

• Boren, T., & Ramey, J. (2000). Thinking aloud: Reconciling theory and practice. IEEE Transactions on Professional Communication, 43, 261–278.
   Web of Science ®Google Scholar
• Bowers, V. A., & Snyder, H. L. (1990). Concurrent versus retrospective verbal protocols for comparing window usability. Proceedings of the Human Factors Society 34th Annual Meeting, 1270–1274.
   Google Scholar
• Bruun, A., Gull, P., Hofmeister, L., & Stage, J. (2009). Let your users do the testing: A comparison of three remote asynchronous usability testing methods. Proceedings of the CHI 2009 Conference on Human Factors in Computing Systems, 1619–1628.
   Google Scholar
• Clemmensen, T., Hertzum, M., Hornbæk, K., Shi, Q., & Yammiyavar, P. (2009). Cultural cognition in usability evaluation. Interacting with Computers, 21, 212–220.
   Web of Science ®Google Scholar
• Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37–46.
   Web of Science ®Google Scholar
• Cooke, L. (2010). Assessing concurrent think-aloud protocol as a usability test method: A technical communication approach. IEEE Transactions on Professional Communication, 53, 202–215.
   Web of Science ®Google Scholar
• Costner, H. L. (1965). Criteria for measures of association. American Sociological Review, 30, 341–353.
   Web of Science ®Google Scholar
• Dumas, J. S., & Fox, J. E. (2008). Usability testing: Current practice and future directions. In A. Sears & J. A. Jacko (Eds.), The human-computer interaction handbook: Fundamentals, evolving technologies, and emerging applications (2nd ed., pp. 1129–1149). New York, NY: Erlbaum.
   Google Scholar
• Dumas, J. S., & Loring, B. (2008). Moderating usability tests: Principles & practices for interacting. Burlington, MA: Morgan Kaufmann.
   Google Scholar
• Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis: Verbal reports as data. Revised edition. Cambridge, MA: MIT Press.
   Google Scholar
• Fox, M. C., Ericsson, K. A., & Best, R. (2011). Do procedures for verbal reporting of thinking have to be reactive? A meta-analysis and recommendations for best reporting methods. Psychological Bulletin, 137, 316–344.
   PubMed Web of Science ®Google Scholar
• Gilhooly, K. J., Fioratou, E., & Henretty, N. (2010). Verbalization and problem solving: Insight and spatial factors. British Journal of Psychology, 101, 81–93.
   PubMed Web of Science ®Google Scholar
• Greenberg, S., & Buxton, B. (2008). Usability evaluation considered harmful (some of the time). Proceedings of the CHI Conference on Human Factors in Computing Systems, 111–120.
   Google Scholar
• Hertzum, M., Hansen, K. D., & Andersen, H. H. K. (2009). Scrutinising usability evaluation: Does thinking aloud affect behaviour and mental workload? Behaviour & Information Technology, 28, 165–181.
   Web of Science ®Google Scholar
• Hertzum, M., & Holmegaard, K. D. (2015). Thinking aloud influences perceived time. Human Factors, 57, 101–109.
   PubMed Web of Science ®Google Scholar
• Hertzum, M., Molich, R., & Jacobsen, N. E. (2014). What you get is what you see: Revisiting the evaluator effect in usability tests. Behaviour & Information Technology, 33, 143–161.
   Web of Science ®Google Scholar
• Hornbæk, K., & Frøkjær, E. (2005). Comparing usability problems and redesign proposals as input to practical systems development. Proceedings of the CHI 2005 Conference on Human Factors in Computing Systems, 391–400.
   Google Scholar
• Lazar, J., Feng, J. H., & Hochheiser, H. (2010). Research methods in human-computer interaction. Chichester, UK: Wiley.
   Google Scholar
• Lewis, J. R. (2014). Usability: Lessons learned … and yet to be learned. International Journal of Human-Computer Interaction, 30, 663–684.
   Web of Science ®Google Scholar
• Liu, D., Bias, R. G., Lease, M., & Kuipers, R. (2012). Crowdsourcing for usability testing. In ASIST2012: Proceedings of the American Society for Information Science and Technology, 1–10.
   Google Scholar
• McDonald, S., Edwards, H. M., & Zhao, T. (2012). Exploring think-alouds in usability testing: An international survey. IEEE Transactions on Professional Communication, 55, 2–19.
   Web of Science ®Google Scholar
• McDonald, S., Zhao, T., & Edwards, H. M. (2013). Dual verbal elicitation: The complementary use of concurrent and retrospective reporting within a usability test. International Journal of Human-Computer Interaction, 29, 647–660.
   Web of Science ®Google Scholar
• Nelson, E. T., & Stavrou, A. (2011). Advantages and disadvantages of remote asynchronous usability testing using Amazon mechanical turk. Proceedings of the Human Factors and Ergonomics Society 55th Annual Meeting, 1080–1084.
   Google Scholar
• Nielsen, J. (1993). Usability engineering. Boston, MA: Academic Press.
   Google Scholar
• Nørgaard, M., & Hornbæk, K. (2006). What do usability evaluators do in practice? An explorative study of think-aloud testing. Proceedings of the Sixth DIS Conference on Designing Interactive Systems, 209–218.
   Google Scholar
• Ohnemus, K. R., & Biers, D. W. (1993). Retrospective versus concurrent thinking-out-loud in usability testing. Proceedings of the Human Factors and Ergonomics Society 37th Annual Meeting, 1127–1131.
   Google Scholar
• Rubin, J., & Chisnell, D. (2008). Handbook of usability testing: How to plan, design, and conduct effective tests (2nd ed.). Indianapolis, IN: Wiley.
   Google Scholar
• Spool, J. M., Scanlon, T., Schroeder, W., Snyder, C., & DeAngelo, T. (1999). Web site usability: A designer’s guide. San Francisco, CA: Morgan Kaufmann.
   Google Scholar
• Van den Haak, M. J., de Jong, M. D. T., & Schellens, P. J. (2003). Retrospective vs. concurrent think-aloud protocols: Testing the usability of an online library catalogue. Behaviour & Information Technology, 22, 339–351.
   Web of Science ®Google Scholar
• Van den Haak, M. J., de Jong, M. D. T., & Schellens, P. J. (2004). Employing think-aloud protocols and constructive interaction to test the usability of online library catalogues: A methodological comparison. Interacting with Computers, 16, 1153–1170.
   Web of Science ®Google Scholar
• Van den Haak, M. J., de Jong, M. D. T., & Schellens, P. J. (2006). Constructive interaction: An analysis of verbal interaction in a usability setting. IEEE Transactions on Professional Communication, 49, 311–324.
   Google Scholar
• Venturi, G., Troost, J., & Jokela, T. (2006). People, organizations, and processes: An inquiry into the adoption of user-centred design in industry. International Journal of Human–Computer Interaction, 21, 219–238.
   Web of Science ®Google Scholar
• Wilson, T. D., & Schooler, J. W. (1991). Thinking too much: Introspection can reduce the quality of preferences and decisions. Journal of Personality and Social Psychology, 60, 181–192.
   PubMed Web of Science ®Google Scholar
• Zhao, T., & McDonald, S. (2010). Keep talking: An analysis of participant utterances gathered using two concurrent think-aloud methods. Proceedings of the NordiCHI2010 Conference on Human–Computer Interaction, 581–590.
   Google Scholar
• Zhao, T., McDonald, S., & Edwards, H. M. (2014). The impact of two different think-aloud instructions in a usability test: A case of just following orders? Behaviour & Information Technology, 33, 163–183.
   Web of Science ®Google Scholar