Advanced search
Publication Cover
Behaviour & Information Technology Volume 28, 2009 - Issue 2
Submit an article Journal homepage
1,220
Views
82
CrossRef citations to date
0
Altmetric
Usability evaluation

Scrutinising usability evaluation: does thinking aloud affect behaviour and mental workload?

Morten Hertzum Computer Science, Roskilde University, Roskilde, DenmarkCorrespondence[email protected]
,
Kristin D. Hansen Computer Science, Roskilde University, Roskilde, Denmark; Systems Analysis Department, Ris⊘ National Laboratory, Roskilde, Denmark
&
Hans H.K. Andersen Systems Analysis Department, Ris⊘ National Laboratory, Roskilde, Denmark
Pages 165-181 | Received 01 Sep 2006, Accepted 01 Oct 2007, Published online: 23 Feb 2009

Citations (82)

Keep up to date with the latest research on this topic with citation updates for this article.

Subscribe to citation updates

Read on this site (15)

Obead Alhadreti. (2022) A Comparison of Synchronous and Asynchronous Remote Usability Testing Methods. International Journal of Human–Computer Interaction 38:3, pages 289-297.
Read now
Obead Alhadreti. (2021) Comparing Two Methods of Usability Testing in Saudi Arabia: Concurrent Think-Aloud vs. Co-Discovery. International Journal of Human–Computer Interaction 37:2, pages 118-130.
Read now
Laura Swettenham, Martin Eubank, Doyeon Won & Amy E Whitehead. (2020) Investigating stress and coping during practice and competition in tennis using think aloud. International Journal of Sport and Exercise Psychology 18:2, pages 218-238.
Read now
Xiang Ji & Pei-Luen Patrick Rau. (2019) A comparison of three think-aloud protocols used to evaluate a voice intelligent agent that expresses emotions. Behaviour & Information Technology 38:4, pages 375-383.
Read now
Leanne M. Willis & Sharon McDonald. (2016) Retrospective protocols in usability testing: a comparison of Post-session RTA versus Post-task RTA reports. Behaviour & Information Technology 35:8, pages 628-643.
Read now
Morten Hertzum, Pia Borlund & Kristina B. Kristoffersen. (2015) What Do Thinking-Aloud Participants Say? A Comparison of Moderated and Unmoderated Usability Sessions. International Journal of Human–Computer Interaction 31:9, pages 557-570.
Read now
Chang-Ping Hu, Wei-Wei Yan & Yuan Hu. (2015) User satisfaction evaluation of microblogging services in China: using the tetra-class model. Behaviour & Information Technology 34:1, pages 17-32.
Read now
James R. Lewis. (2014) Usability: Lessons Learned … and Yet to Be Learned. International Journal of Human–Computer Interaction 30:9, pages 663-684.
Read now
Tingting Zhao, Sharon McDonald & Helen M. Edwards. (2014) The impact of two different think-aloud instructions in a usability test: a case of just following orders?. Behaviour & Information Technology 33:2, pages 163-183.
Read now
Sharon McDonald, Tingting Zhao & Helen M. Edwards. (2013) Dual Verbal Elicitation: The Complementary Use of Concurrent and Retrospective Reporting Within a Usability Test. International Journal of Human–Computer Interaction 29:10, pages 647-660.
Read now
Morten Hertzum & Kristin Due Holmegaard. (2013) Thinking Aloud in the Presence of Interruptions and Time Constraints. International Journal of Human–Computer Interaction 29:5, pages 351-364.
Read now
Alan Woolrych, Kasper Hornbæk, Erik Frøkjær & Gilbert Cockton. (2011) Ingredients and Meals Rather Than Recipes: A Proposal for Research That Does Not Treat Usability Evaluation Methods as Indivisible Wholes. International Journal of Human–Computer Interaction 27:10, pages 940-970.
Read now
Lex van Velsen, Thea van der Geest & Rob Klaassen. (2011) Identifying Usability Issues for Personalization During Formative Evaluations: A Comparison of Three Methods. International Journal of Human–Computer Interaction 27:7, pages 670-698.
Read now
Petter Bae Brandtzæg, Marika Lüders & Jan Håvard Skjetne. (2010) Too Many Facebook “Friends”? Content Sharing and Sociability Versus the Need for Privacy in Social Network Sites. International Journal of Human–Computer Interaction 26:11-12, pages 1006-1030.
Read now
GREGORY SCHRAW. (2010) Measuring Self-Regulation in Computer-Based Learning Environments. Educational Psychologist 45:4, pages 258-266.
Read now

Articles from other publishers (67)

Arpit Bhatia, Kasper Hornbæk & Hasti Seifi. (2024) Augmenting the feel of real objects: An analysis of haptic augmented reality. International Journal of Human-Computer Studies 185, pages 103244.
Crossref
Paweł Weichbroth. (2024) Usability Testing of Mobile Applications: A Methodological Framework. Applied Sciences 14:5, pages 1792.
Crossref
Elien Goossens. 2024. Understanding Prisoner Victimisation. Understanding Prisoner Victimisation 143 173 .
Jessie Mitchell, Kelly Clanchy & Camila Shirota. (2023) Towards Translation of Novel Neurorehabilitation Systems: A Practical Approach to Usability Testing. Towards Translation of Novel Neurorehabilitation Systems: A Practical Approach to Usability Testing.
Snehal Prabhudesai, Leyao Yang, Sumit Asthana, Xun Huan, Q. Vera Liao & Nikola Banovic. (2023) Understanding Uncertainty: How Lay Decision-makers Perceive and Interpret Uncertainty in Human-AI Decision Making. Understanding Uncertainty: How Lay Decision-makers Perceive and Interpret Uncertainty in Human-AI Decision Making.
Ramzi Abou Chahine, Sid Padmanabhuni, Pooyan Fazli & Hasti Seifi. (2023) Designing and Evaluating Interactive Tools for a Robot Hand Collection. Designing and Evaluating Interactive Tools for a Robot Hand Collection.
Gisela Reyes-Cruz, Joel E. Fischer & Stuart Reeves. (2022) Demonstrating Interaction: The Case of Assistive Technology. ACM Transactions on Computer-Human Interaction 29:5, pages 1-37.
Crossref
Wencheng Su, Zhangping Lu, Yinglin Sun & Guifeng Liu. (2021) Let eyes tell: experimental research on university library signage system and users' wayfinding behavior. Library Hi Tech 40:1, pages 198-221.
Crossref
Jeffrey A. Greene, Rebekah F. Duke, Rebekah Freed, Dalila Dragnić-Cindrić & Brian M. Cartiff. (2022) Effects of an ego-depletion intervention upon online learning. Computers & Education 177, pages 104362.
Crossref
Laura Swettenham & Amy E. Whitehead. (2022) Developing the Triad of Knowledge in Coaching: Think Aloud as a Reflective Tool Within a Category 1 Football Academy. International Sport Coaching Journal 9:1, pages 122-132.
Crossref
Alessia Eletta Coppi, Catharine Oertel & Alberto Cattaneo. (2021) Effects of Experts’ Annotations on Fashion Designers Apprentices’ Gaze Patterns and Verbalisations. Vocations and Learning 14:3, pages 511-531.
Crossref
James R. Lewis & Jeff Sauro. 2021. HANDBOOK OF HUMAN FACTORS AND ERGONOMICS. HANDBOOK OF HUMAN FACTORS AND ERGONOMICS 972 1015 .
Anders Bruun, Effie Lai-Chong Law, Thomas Dyhre Nielsen & Matthias Heintz. (2021) Do You Feel the Same? On the Robustness of Cued-Recall Debriefing for User Experience Evaluation. ACM Transactions on Computer-Human Interaction 28:4, pages 1-45.
Crossref
Rebecca Krosnick, Fraser Anderson, Justin Matejka, Steve Oney, Walter S. Lasecki, Tovi Grossman & George Fitzmaurice. (2021) Think-Aloud Computing: Supporting Rich and Low-Effort Knowledge Capture. Think-Aloud Computing: Supporting Rich and Low-Effort Knowledge Capture.
Sharon McDonald, Gilbert Cockton & Alastair Irons. (2020) The Impact of Thinking-Aloud on Usability Inspection. Proceedings of the ACM on Human-Computer Interaction 4:EICS, pages 1-22.
Crossref
Anna K. Trukenbrod, Nils Backhaus & Roland Thomaschke. (2020) Measuring subjectively experienced time in usability and user experience testing scenarios. International Journal of Human-Computer Studies 138, pages 102399.
Crossref
Bogeum Choi, Austin Ward, Yuan Li, Jaime Arguello & Robert Capra. (2019) The Effects of Task Complexity on the Use of Different Types of Information in a Search Assistance Tool. ACM Transactions on Information Systems 38:1, pages 1-28.
Crossref
Martina Breuning, Lisa Schäfer-Fauth, Gabriele Lucius-Hoene & Christine Holmberg. (2020) Connecting one’s own illness story to the illness experiences of others on a website—An evaluation study using the think aloud method. Patient Education and Counseling 103:1, pages 199-207.
Crossref
Morten Hertzum. 2020. Usability Testing. Usability Testing.
Obead Alhadreti. 2020. Learning and Collaboration Technologies. Designing, Developing and Deploying Learning Experiences. Learning and Collaboration Technologies. Designing, Developing and Deploying Learning Experiences 3 19 .
Tania Calle-Jimenez, Doris Tutillo-Sanchez, Sandra Sanchez-Gordon, Janio Jadán-Guerrero, Cesar Guevara, Patricio Lara-Alvarez, Patricia Acosta-Vargas, Luis Salvador-Ullauri & Isabel L. Nunes. 2020. Advances in Human Factors and Systems Interaction. Advances in Human Factors and Systems Interaction 361 373 .
Chamila Wijayarathna & Nalin Asanka Gamagedara Arachchilage. (2019) Using cognitive dimensions to evaluate the usability of security APIs: An empirical investigation. Information and Software Technology 115, pages 5-19.
Crossref
Mingming Fan, Jinglan Lin, Christina Chung & Khai N. Truong. (2019) Concurrent Think-Aloud Verbalizations and Usability Problems. ACM Transactions on Computer-Human Interaction 26:5, pages 1-35.
Crossref
Marcio Moreno, Rafael Brandão, Rodrigo Santos, Wallas Sousa & Renato Cerqueira. (2019) Representing Experts' Interpretive Trails with Hyperknowledge Specifications. Representing Experts' Interpretive Trails with Hyperknowledge Specifications.
Stuart Reeves. (2019) How UX Practitioners Produce Findings in Usability Testing. ACM Transactions on Computer-Human Interaction 26:1, pages 1-38.
Crossref
Ali Kürşat Erümit, İsmail Çetin, Mehmet Kokoç, Temel Kösa, Vasif Nabiyev & Emine Selin Aygün. (2019) Designing a Usibility Assessment Process for Adaptive Intelligent Tutoring Systems: A Case Study. Turkish Online Journal of Qualitative Inquiry, pages 141-179.
Crossref
Mingming Fan, Ke Wu, Jian Zhao, Yue Li, Winter Wei & Khai N. Truong. (2019) VisTA: Integrating Machine Intelligence with Visualization to Support the Investigation of Think-Aloud Sessions. IEEE Transactions on Visualization and Computer Graphics, pages 1-1.
Crossref
Gencay Deniz & Pınar Onay Durdu. (2019) A comparison of mobile form controls for different tasks. Computer Standards & Interfaces 61, pages 97-106.
Crossref
Günter Alce, Eva-Maria Ternblad & Mattias Wallergård. 2019. Human-Computer Interaction – INTERACT 2019. Human-Computer Interaction – INTERACT 2019 267 286 .
Josefine Karlsson, Paul van den Broek, Anne Helder, Marian Hickendorff, Arnout Koornneef & Linda van Leijenhorst. (2018) Profiles of young readers: Evidence from thinking aloud while reading narrative and expository texts. Learning and Individual Differences 67, pages 105-116.
Crossref
Morten Hertzum & Kristina Bonde Kristoffersen. (2018) What do usability test moderators say?. What do usability test moderators say?.
Ladislao Salmerón, Helge I. Strømsø, Yvonne Kammerer, Marc Stadtler & Paul van den Broek. 2018. Learning to Read in a Digital World. Learning to Read in a Digital World 91 120 .
Weipeng Chen, Tao Lin, Li Chen & Peisa Yuan. (2018) Automated comprehensive evaluation approach for user interface satisfaction based on concurrent think-aloud method. Universal Access in the Information Society 17:3, pages 635-647.
Crossref
Sara Nygårdhs, Christer Ahlström, Jonas Ihlström & Katja Kircher. (2018) Bicyclists’ adaptation strategies when interacting with text messages in urban environments. Cognition, Technology & Work 20:3, pages 377-388.
Crossref
Daniel Reinhardt, Jeremias Kuge & Jörn Hurtienne. 2018. Design, User Experience, and Usability: Theory and Practice. Design, User Experience, and Usability: Theory and Practice 528 545 .
Simon Laing, Mark Apperley & Masood Masoodian. (2017) Investigating the effects of client imagery on the ideation process of graphic design. Design Studies 53, pages 78-98.
Crossref
Daniel Reinhardt & Jörn Hurtienne. (2017) Interaction Under Pressure. Interaction Under Pressure.
M. Karimi, J. Brazier & S. Paisley. (2017) How do individuals value health states? A qualitative investigation. Social Science & Medicine 172, pages 80-88.
Crossref
Rafael R. M. Brandão, Marcio F. Moreno & Renato F. G. Cerqueira. 2017. Universal Access in Human–Computer Interaction. Design and Development Approaches and Methods. Universal Access in Human–Computer Interaction. Design and Development Approaches and Methods 291 300 .
Haakon Lund. (2016) Eye tracking in library and information science: a literature review. Library Hi Tech 34:4, pages 585-614.
Crossref
Anna L. Toscano-Zapién, Daniel Velázquez-López & David N. Velázquez-Martínez. (2016) Attentional Mechanisms during the Performance of a Subsecond Timing Task. PLOS ONE 11:7, pages e0158508.
Crossref
Sunmoo Yoon, Bevin Cohen, Kenrick D. Cato, Jianfang Liu & Elaine L. Larson. (2016) Visualization of Data Regarding Infections Using Eye Tracking Techniques. Journal of Nursing Scholarship 48:3, pages 244-253.
Crossref
Sharon McDonald, Tingting Zhao & Helen M. Edwards. (2016) Look Who's Talking: Evaluating the Utility of Interventions During an Interactive Think-Aloud. Interacting with Computers 28:3, pages 387-403.
Crossref
Morten Hertzum. (2016) A usability test is not an interview. Interactions 23:2, pages 82-84.
Crossref
Rakefet Ackerman, Avi Parush, Fareda Nassar & Avraham Shtub. (2016) Metacognition and system usability: Incorporating metacognitive research paradigm into usability testing. Computers in Human Behavior 54, pages 101-113.
Crossref
Kimmo Tarkkanen & Ville Harkke. 2016. Technology and Intimacy: Choice or Coercion. Technology and Intimacy: Choice or Coercion 215 228 .
Margaret Varga & Caroline Varga. 2016. Building Trust in Information. Building Trust in Information 141 150 .
Victoria L. Lemieux. 2016. Building Trust in Information. Building Trust in Information 3 45 .
Thiago Adriano Coleti, Marcelo Morandini, Pedro Luiz Pizzigatti Correa, Daniel Lins da Silva & Clodis Boscarioli. (2015) Using keywords to support the verbalization in usability evaluation. Using keywords to support the verbalization in usability evaluation.
Morten Hertzum & Kristin Due Holmegaard. (2014) Thinking Aloud Influences Perceived Time. Human Factors: The Journal of the Human Factors and Ergonomics Society 57:1, pages 101-109.
Crossref
Beth Burbach, Susan Barnason & Sarah A Thompson. (2015) Using “Think Aloud” to Capture Clinical Reasoning during Patient Simulation. International Journal of Nursing Education Scholarship 12:1, pages 1-7.
Crossref
Anders Bruun & Jan Stage. 2015. Human-Computer Interaction – INTERACT 2015. Human-Computer Interaction – INTERACT 2015 159 176 .
Matthew F. Pike, Horia A. Maior, Martin Porcheron, Sarah C. Sharples & Max L. Wilson. (2014) Measuring the effect of think aloud protocols on workload using fNIRS. Measuring the effect of think aloud protocols on workload using fNIRS.
Leo Lentz & Sanne Elling. 2014. Evaluating Websites and Web Services. Evaluating Websites and Web Services 95 117 .
Thiago Adriano Coleti, Marcelo Morandini & Fátima de Lourdes dos Santos Nunes. 2014. Human-Computer Interaction. Theories, Methods, and Tools. Human-Computer Interaction. Theories, Methods, and Tools 554 564 .
Ana Oliveira, Cátia Pinho, Sandra Monteiro, Ana Marcos & Alda Marques. (2013) Usability testing of a respiratory interface using computer screen and facial expressions videos. Computers in Biology and Medicine 43:12, pages 2205-2213.
Crossref
Sharon McDonald, Kenneth McGarry & Leanne M. Willis. (2013) Thinking-aloud about web navigation. Proceedings of the Human Factors and Ergonomics Society Annual Meeting 57:1, pages 2037-2041.
Crossref
Sharon McDonald & Helen Petrie. (2013) The effect of global instructions on think-aloud testing. The effect of global instructions on think-aloud testing.
Amy M. Gill & Blair Nonnecke. (2012) Think aloud. Think aloud.
Sanne Elling, Leo Lentz & Menno de Jong. (2012) Combining Concurrent Think-Aloud Protocols and Eye-Tracking Observations: An Analysis of Verbalizations and Silences. IEEE Transactions on Professional Communication 55:3, pages 206-220.
Crossref
Sharon McDonald, Helen M. Edwards & Tingting Zhao. (2012) Exploring Think-Alouds in Usability Testing: An International Survey. IEEE Transactions on Professional Communication 55:1, pages 2-19.
Crossref
Sanne Elling, Leo Lentz & Menno de Jong. (2011) Retrospective think-aloud method. Retrospective think-aloud method.
Alexandros Paramythis, Stephan Weibelzahl & Judith Masthoff. (2010) Layered evaluation of interactive adaptive systems: framework and formative methods. User Modeling and User-Adapted Interaction 20:5, pages 383-453.
Crossref
Tingting ZhaoSharon McDonald. (2010) Keep talking. Keep talking.
Lynne Cooke. (2010) Assessing Concurrent Think-Aloud Protocol as a Usability Test Method: A Technical Communication Approach. IEEE Transactions on Professional Communication 53:3, pages 202-215.
Crossref
Erica L. Olmsted-Hawala, Elizabeth D. Murphy, Sam Hawala & Kathleen T. Ashenfelter. (2010) Think-aloud protocols: Analyzing three different think-aloud protocols with counts of verbalized frustrations in a usability study of an information-rich Web site. Think-aloud protocols: Analyzing three different think-aloud protocols with counts of verbalized frustrations in a usability study of an information-rich Web site.
Erica L. Olmsted-Hawala, Elizabeth D. Murphy, Sam Hawala & Kathleen T. Ashenfelter. (2010) Think-aloud protocols. Think-aloud protocols.

Reprints and Corporate Permissions

Please note: Selecting permissions does not provide access to the full text of the article, please see our help page How do I view content?

To request a reprint or corporate permissions for this article, please click on the relevant link below:

Order Reprints Request Corporate Permissions

Academic Permissions

Please note: Selecting permissions does not provide access to the full text of the article, please see our help page How do I view content?

Obtain permissions instantly via Rightslink by clicking on the button below:

Request Academic Permissions

If you are unable to obtain permissions via Rightslink, please complete and submit this Permissions form. For more information, please visit our Permissions help page.

Related research

People also read lists articles that other readers of this article have read.

Recommended articles lists articles that we recommend and is powered by our AI driven recommendation engine.

Cited by lists all citing articles based on Crossref citations.
Articles with the Crossref icon will open in a new tab.