Justify Your Answer: The Role of Written Think Aloud in Script Concordance Testing

ABSTRACT

Construct: Clinical reasoning assessment is a growing area of interest in the medical education literature. Script concordance testing (SCT) evaluates clinical reasoning in conditions of uncertainty and has emerged as an innovative tool in the domain of clinical reasoning assessment. SCT quantifies the degree of concordance between a learner and an experienced clinician and attempts to capture the breadth of responses of expert clinicians, acknowledging the significant yet acceptable variation in practice under situations of uncertainty. Background: SCT has been shown to be a valid and reliable clinical reasoning assessment tool. However, as SCT provides only quantitative information, it may not provide a complete assessment of clinical reasoning. Approach: Think aloud (TA) is a qualitative research tool used in clinical reasoning assessment in which learners verbalize their thought process around an assigned task. This study explores the use of TA, in the form of written reflection, in SCT to assess resident clinical reasoning, hypothesizing that the information obtained from the written TA would enrich the quantitative data obtained through SCT. Ninety-one pediatric postgraduate trainees and 21 pediatricians from 4 Canadian training centers completed an online test consisting of 24 SCT cases immediately followed by retrospective written TA. Six of 24 cases were selected to gather TA data. These cases were chosen to allow all phases of clinical decision making (diagnosis, investigation, and treatment) to be represented in the TA data. Inductive thematic analysis was employed when systematically reviewing TA responses. Results: Three main benefits of adding written TA to SCT were identified: (a) uncovering instances of incorrect clinical reasoning despite a correct SCT response, (b) revealing sound clinical reasoning in the context of a suboptimal SCT response, and (c) detecting question misinterpretation. Conclusions: Written TA can optimize SCT by demonstrating when correct examinee responses are based on guessing or uncertainty rather than robust clinical rationale. TA can also enhance SCT by allowing examinees to provide justification for responses that otherwise would have been considered incorrect and by identifying questions that are frequently misinterpreted to avoid including them in future examinations. TA also has significant value in differentiating between acceptable variations in expert clinician responses and deviance associated with faulty rationale or question misinterpretation; this could improve SCT reliability. A written TA protocol appears to be a valuable tool to assess trainees' clinical reasoning and can strengthen the quantitative assessment provided by SCT.

KEYWORDS:

• Clinical reasoning
• assessment
• think aloud
• script concordance testing
• postgraduate medical education

Acknowledgments

We thank all the study participants for their time, as well as the pediatric residency training programs at the University of Alberta, University of British Columbia, University of Calgary, and the University of Saskatchewan for their support of this project. We also express our gratitude to Ms. Linda Beatty for her assistance with preparation of this manuscript.

Funding

This research was made possible through funds provided by the Royal College of Physicians and Surgeons of Canada as part of a Fellowship for Studies in Medical Education, received by Dr. Suzette Cooke (2012).