Twelve tips for applying the think-aloud method to capture cognitive processes

Abstract

The think-aloud method is an established technique for studying human thought (cognitive) processes. Problem-solving and decision-making are essential skills for medical professionals, and the cognitive processes underlying these skills are complex. Studying these thought processes would enable educators, clinicians, and researchers to modify or refine their approaches and interventions. The think-aloud method has been utilized for capturing cognitive processes in a variety of fields, including computer usability, sports and cognitive psychology. Medical education also recognizes thought processes as valuable data for research and education. This article aims to guide researchers and educators through the preparation and implementation of a think-aloud method to record participants’ thought processes during an activity.

Keywords:

• Think-aloud method
• learning
• cognitive processes
• verbalizations

Introduction

The think-aloud (TA) method studies cognitive processes and decision-making strategies by having people voice their thoughts while performing a task or solving a problem (Ericsson and Simon 1980). Ericsson and Simon’s theoretical stance for the think-aloud approach is based on the distinction between working and long-term memory. This theory states that concurrent reasoning occurs in working memory, which is a cognitive system of limited capacity that temporarily holds information for immediate processing (Atkinson and Shiffrin 1968). The literature suggests that we cannot directly access higher-order mental processes like evaluation, judgement, problem-solving, and behavior initiation (Nisbett and Wilson 1977). However, verbal reports given during or immediately after these mental processes may reflect conscious cognition (Charters 2003). The two most common verbal reporting methods are concurrent and retrospective think-aloud. In retrospective think-aloud, unlike concurrent, participants verbalize their thoughts after completing a specified task. They do so by viewing a video recording of themselves performing the task (Van Den Haak et al. 2003). Thus, think-aloud can provide insight into many real-time cognitive processes that occur in working memory. Analysing these verbalizations can help researchers and healthcare professionals gain insights into the mental processes involved, which can be used to develop strategies for improving decision-making, patient care, and education.

In general, medical education research focuses on studying the relationship between interventions and their outcomes in a wide range of teaching-learning contexts (Prystowsky and Bordage 2001; Emery et al. 2022). While such outcomes are extremely important, the think-aloud method presents an opportunity to explore an area that has been largely overlooked but would significantly enhance our understanding of the cognitive processes that underlie the effects of an intervention. Examples of such encounters include thought processes during clinical decision-making, problem-solving, clinical reasoning, expertise development and performance monitoring. (Tower et al. 2019; Altalhi et al. 2021; van de Pol et al. 2021). In the context of education and training, gaining access to an expert’s thought processes during clinical decision-making or obtaining insights into the mental processes of novice learners to support them with effective feedback can enhance students’ learning experiences (Lundgrén-Laine and Salanterä 2010).

Using the think-aloud technique for research can be time-consuming and laborious (Barkaoui 2011). The following tips could assist researchers in preparing and implementing the think-aloud technique. These tips combine information gleaned from the literature and the author’s own experiences.

Tip 1

Ensure methodological appropriateness

It is essential to carefully consider the research question, context, resources, and expertise of the researcher before selecting an appropriate method for capturing the participant’s thought processes (Wolcott and Lobczowski 2021). The concurrent think-aloud technique is ideal for collecting verbal data on the thought processes of nonverbal tasks (Johnson et al. 2022). This technique can be used to understand the cognitive processes behind visual interpretations, such as reviewing an MRI report (Derrett et al. 2009), inspecting the retina using an ophthalmoscope, or interpreting an X-ray of the chest (Van der Gijp et al. 2014). It can also be applied to studying specific task management, such as performing intravitreal injections to treat retinal conditions or performing a colonoscopy (Johnson et al. 2022). Additionally, it can be used to gather data on user experiences of electronic applications (Jaspers et al. 2004) including electronic clinical decision support systems, digital health promotion applications (Miller et al. 2020), and electronic medical records (Alshehri and Alanazi 2023). Furthermore, it can be employed to investigate the impact of pedagogical interventions, such as the learning influence of concept mapping (Hilbert and Renkl 2008) or enhancing clinical reasoning skills among students (Tedesco-Schneck 2019). On the other hand, retrospective think-aloud is beneficial for tasks that require verbal communication or have a high complexity, which would make concurrent verbalization difficult and may affect the quality of data. For instance, if the task under consideration is a teaching session (Lingemann et al. 2012), eye-tracking studies that require the respondent to describe the reasons for gazing at certain points (cued-retrospective think-aloud) (Al-Moteri et al. 2022), therapy sessions that require patient interactions (Kleynen et al. 2017), or studies involving timed activities (Surry et al. 2017), a retrospective verbalization would be more appropriate. Between the two, concurrent think-aloud is the better strategy for capturing ongoing thought processes in working memory. Nonetheless, retrospective think-aloud can supplement concurrent data, as it provides a more thorough description of thought processes because it allows for queries and clarifications.

Tip 2

Sample purposefully

Think-aloud is a qualitative approach that gathers extensive participant data, like a ‘collective case-study’ where participants’ cognitive processes are extracted and analyzed (Charters 2003). Typically, researchers calculate sample size to estimate the number of participants required to reach ‘theoretical sufficiency,’ the point at which incoming data no longer provides valuable information regarding the research question. Qualitative studies have observed that about 90% of all concepts were discovered in 6 to 18 interviews (Morgan et al. 2002; Guest et al. 2006; Francis et al. 2010; Namey et al. 2016; Guest et al. 2020). Think-aloud studies reveal that it can yield meaningful results with as few as five participants (Virzi 1992). These findings suggest the range of minimum sample size one should aim for in qualitative research, and the think-aloud technique may adopt a similar approach. After acquiring the initial dataset, researchers can analyse it in parallel and stop recruiting participants once they have reached a sufficient understanding of their research question.

The sampling for the think-aloud method should be purposeful (Seo et al. 2022), representative, and as heterogeneous as possible (Wolcott and Lobczowski 2021) to obtain a sample that reveals all the dimensions of the phenomenon under investigation. It is advisable to refrain from including domain experts as participants as their expertise with the task may result in incomplete verbalizations (Van Someren et al. 1994). The participants’ verbalization abilities are an additional factor that may be considered, as it may impact the completeness of the data (Van Someren et al. 1994; Vanicek and Popelka 2023).

Tip 3

Ensure a quiet and controlled task setting

Think-aloud sessions are always conducted individually. When preparing the setting for think-aloud sessions, it is necessary to provide a quiet, soundproof environment and a comfortable seating. Normally, a think-aloud session would be lengthy and exhausting for the participant. Providing a controlled environment is essential to encourage them to think aloud naturally (Johnson et al. 2022). It is unlikely that the participant will feel completely comfortable thinking aloud in a place surrounded by people. Preferably, a single observer who is unknown to the research objectives but well versed with the think-aloud procedure is present during the session and interacts (as necessary and minimally) with the participant. While planning for an online think-aloud session, a non-interruptive task setting could be accomplished by instructing participants to be in a quiet, private area of their choosing. If we are not interested to document any non-verbal cues, we may give the participants an option to keep their video off while they think aloud. In our experience of conducting online think-aloud sessions, we have observed that when participants realize they are alone in a comfortable, quiet environment, they quickly switch into a natural think-aloud mode.

Tip 4

Provide limited but precise instructions to the participant

The basic act of thinking aloud does not interfere with the task performance or the cognitive processes (Ericsson and Simon 1993). The think-aloud instructions for the participant should be simple and straightforward. It could read as follows: ‘Try to say everything that comes to your mind while you engage with the task’ However, talking while performing a task is difficult and not everyone is naturally capable of doing so. In order to achieve the desired outcome, the participant must fully understand the think aloud process. A few complementary instructions could potentially enhance the data’s validity. Ericsson and Simon (1993) recommend the following additional instructions: a) to speak all thoughts, even if they are unrelated to the task; b) to refrain from explaining the thoughts; c) to not try to plan out what to say; d) to imagine the participants are alone and speak to themselves; and e) to speak continuously. These additional instructions facilitate the participants’ understanding of the think-aloud procedure, resulting in a quality output. Researchers frequently make the error of asking participants to describe or explain their thoughts or actions. Such instructions which stimulate the participant to make interpretations result in additional cognitive processes, which may compromise the validity of the thought processes collected (Fox et al. 2011).

Tip 5

Ensure a domain-specific warm-up training

It is important to include think-aloud training prior to the task, as this provides participants with the confidence necessary to verbalize their thoughts (Ericsson and Simon 1993). To practice thinking aloud, Ericsson and Simon suggest incorporating basic mental arithmetic (e.g. 23 × 34 or 342 + 54) and a problem-solving task (e.g. solving an anagram). However, Van Someren et al. (1994) advised choosing a practice TA task that is not too dissimilar from the target task (domain-specific task). These training sessions permit the experimenter to observe and correct the participant’s think-aloud performance as needed.

Think-aloud tasks are cognitively challenging due to the combination of thinking and voicing. In general, the energy required to execute each individual task limits the number of tasks that can be performed concurrently (Schaefer 2014). Therefore, a well-rehearsed task requires less cognitive effort (i.e. automatic processing) than a novel task (i.e. controlled processing). Accordingly, certain researchers provided participants with TA training that included both domain-specific and general instruction. Whitehead et al. (2019) provided participants with TA training that included both domain-specific and general instruction a week before the actual task. Prior to an online think-aloud session during the pandemic, Noushad et al. (2023) shared with participants a video outlining the TA instructions and examples of thinking aloud, which was then followed by both generic and domain-specific TA practice tasks on the day of the TA session. An advantage of pre-recorded instructions is that all participants receive identical pre-task training.

Tip 6

Refrain from interrupting or prompting the participant

When the participant is performing the task, the observer’s role is limited. They must position themselves at a suitable distance behind the participant to avoid distracting them, but close enough to hear their verbalisations (Vanicek and Popelka 2023). The only occasion where the observer should intervene is when the subject stops talking. In that case, the observer should prompt the participant by simply saying, ‘Please keep talking.’ This is sufficient to alert and keep the subject engaged in talking. The literature is mixed about how long the observer needs to wait until they interrupt after the participant has stopped talking: ranging from 5 to 20 s (Nicholls and Polman 2008; Whitehead et al. 2015; Birch and Whitehead 2020). This time gap is mostly dependent on the type of cognitive task that participants are performing. Observers have a challenging job, especially if they know the task domain. They tend to correct or assist subjects. Inexperienced observers interact and explore participant verbalizations to maximize data utility (Van Someren et al. 1994). In fact, this disturbs think-aloud processes and reduces data validity. To avoid needless interference, the observer should have expertise gathering think-aloud data or necessary training.

Tip 7

Capture the verbalizations completely

Think-aloud verbalizations must be recorded with an audio or video recorder and can be supplemented with note taking. Typically, a combination of these options is chosen. Recording methods can be determined based on the participant’s consent. If the researchers are interested in nonverbal behaviour, then video recording is indispensable; taking notes alone will not suffice. It may appear trivial but double-check the functionality of your recording devices is vital. It is advised to record a backup on a second device to avoid losing data.

Tip 8

Transcribe the verbalizations verbatim

Think-aloud verbalizations are transcribed word-for-word (verbatim). Unlike written sentences, TA sentences are unstructured, which makes transcription more difficult. Because the transcriber may encounter uncommon dialects, mumbles and word-finishers, transcription is also time-consuming. If a word, phrase, or sentence is unrecognizable, write 'unclear’ in the transcription. Transcribe everything, including participant instructions, observer interruptions, and out-of-context statements. Typing everything out helps recreating the think-aloud session as close as possible, which is essential for the analysis phase. Documenting stalemates and impasses is essential as well. The transcriptionist must be told even to record the humming, hawing or stammering of the participant. That means it is very important for the transcriptionist to stay as close to the verbal reports as possible. Pauses between words or short pauses are documented with ‘dots.’ ‘I believe … I would score here 70%.’ However, long pauses can be classified as ‘pause.’ Another challenge is choosing punctuation. A sentence that is grammatically correct is simple to manage. However, fully structured sentences are uncommon in TA protocols. The majority of utterances are incomplete or poorly structured. Instead of punctuation, initiate new sentences in these instances.

Tip 9

Allow thinking aloud in participants’ preferred language

Reporting the thought processes during the performance of a task is demanding. Bowles (2010) concluded in her meta-analysis that a number of factors can hinder task performance and affect the quality of verbalizations, including the type of verbal reports, the type of tasks, the cognitive demand of the task, and the language proficiency of the participants. It has been observed that instructing and limiting participants’ verbalization in their second language hinders their natural cognitive processes (Sachs and Polio 2007), as compared to those who were permitted to choose their verbalization language (Yanguas and Lado 2012). In our experience of allowing participants to choose the language of verbalization, we observed them using a mixture of their first and second languages, but their mother language predominated. Therefore, participants should be allowed to freely choose their language and to switch between their native and second languages.

Tip 10

Triangulate to enhance data validity

Ericsson and Simon (1980) remarked that the think-aloud method may not provide enough working memory information because thinking processes are too fast to express. However, immediate follow-up inquiries (retrospective reporting) may capture some working memory information right after a task (Eccles and Arsal 2017). This additional information supplements and expands think-aloud data. However, researchers often forget to focus on task-specific questions while participants reported their thoughts retrospectively. Instead, participants would be given the chance to elaborate on their general thoughts (e.g. ‘What did you think when you first looked at the radiography report?’). It may prompt participants to use long-term memory to supplement working memory. If the immediate retrospective inquiry is limited to the task process, it may provide greater depth to the TA data (Abdel Latif 2019). Such retrospective questioning also lets the researcher clarify unclear utterings made during think aloud (Yiu et al. 2022).

Many other techniques are used along with think-aloud to enhance the validity of data. While investigating self-regulated learning processes, Fan et al. (2023) integrated think aloud data with trace data (data that includes navigational logs such as eye movements, mouse movements and, keystrokes) of online learners. Similarly, the perceptual and attentional processes collected via eye tracking are an important complement to think aloud (Godfroid and Spino 2015). Nonverbal cues such as facial expressions, posture, pauses, silence, speech rate, errors, or misreading may be indicative of participants’ ease, comfort, or working memory load (Charters 2003). To supplement think-aloud data, these non-verbal cues could be recorded or observed (Aitken et al. 2011; Abdulmohdi and Mcvicar 2023). Think aloud data could be combined with questionnaires (Ghenai et al. 2020) and interviews (Kernebeck et al. 2022). However, the research topic determines the data triangulation method(s).

Tip 11

Flexibly analyze the data

Verbal Protocol Analysis (VPA) is the conventional method for examining transcribed think-aloud data (Ericsson and Simon 1993). VPA aims to provide insight into the decision-making processes of participants by tracing their thinking path in three stages: referring phrase analysis, assertion phrase analysis, and script analysis. Referring phrase analysis examines the language and specific phrases used when participants refer to objects or concepts during a task or thought process. Assertion phrase analysis looks at the declarative statements made by participants during a task, which can reveal problem-solving or decision-making strategies. Finally, script analysis identifies the structured sequence of actions, events, or steps followed by participants in their decision-making process (Abdulmohdi and Mcvicar 2023).

The think-aloud technique allows for a variety of qualitative analysis methods to be used, depending on the research question. These methods may include using a predetermined coding framework (deductive approach), systematic coding of emergent idea units (inductive approach), or a combination of both (Al-Moteri et al. 2022). The coded data are then categorized and grouped to identify common patterns or themes, which represent the underlying meaning of the data. Additionally, quantitative analysis can be conducted on the think-aloud data to focus on the distribution, pattern, or frequency of idea units. This flexibility in data analysis makes the think-aloud technique suitable for studying various research questions.

Tip 12

Tackle concerns of reactivity and non-veridicality

Although there is substantial evidence that the think-aloud method is consistently valid, it has some drawbacks. These shortcomings are referred to as 'reactivity’ and 'non-veridicality.' Reporting the thinking process while performing an activity may increase the cognitive load to the point where the participant does not report what they are doing, which is called reactivity (Russo et al. 1989). However, Ericsson and Simon (1998) observed that if the concurrent think-aloud method is combined with precise participant instruction (Tip 4), appropriate training (Tip 6), and minimal researcher interruption (Tip 5), the impact of reactivity can be minimized. A meta-analysis by Fox et al. (2011) confirmed that the effect of reactivity is negligible in studies that use the concurrent think-aloud method. The second concern about the think-aloud method is that verbalizations may not always accurately reflect true cognitive processes, which is referred to as 'non-veridicality’ (Russo et al. 1989). However, researchers have disputed this concern through task analysis (Fox et al. 2011; Ericsson et al. 2018), and more recently, by comparing brain functional magnetic resonance imaging between concurrent think-aloud and silent thinking (Durning et al. 2012). To address the issue of non-veridicality, researchers may consider cross-validating their think-aloud data with observations and retrospective probing. This emphasises the significance of thorough planning and proper implementation of the think-aloud method to obtain valid data.

Conclusion

In medical education, the study of concurrent thought processes during cognitive task performance is an emerging method of investigation. These think-aloud data can be used independently or as a supplement to strengthen the evidence. The think-aloud method is typically arduous for both the researcher and the participants. The key to obtaining valid data from this method is meticulous preparation to ensure the process is less demanding to the participant. We believe that the aforementioned tips will be helpful to those who wish to implement this method, as they are based on a combination of standards reported in the literature and experience.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

The authors of this article received no financial support for the preparation of its content.

Notes on contributors

Babu Noushad

Babu Noushad, MOptom, MHPE, is a faculty at the Department of Optometry, College of Health Sciences, University of Buraimi, Sultanate of Oman. He is currently a PhD scholar in the School of Health Professions Education, Maastricht University, The Netherlands.

Pascal W. M. Van Gerven

Pascal W.M. Van Gerven, PhD, is an associate professor at the Department of Educational Development and Research, School of Health Professions Education, Maastricht University, The Netherlands.

Anique B. H. de Bruin

Anique B.H. de Bruin, PhD, is a professor of educational psychology at the Department of Educational Development and Research, and Vice-director of the School of Health Professions Education, Maastricht University, The Netherlands.