General practice trainees’ learning experiences of formative think-aloud script concordance testing

ABSTRACT

General practitioners (GPs) make timely and accurate clinical decisions, often in the contexts of complexity and uncertainty. However, structured approaches to facilitate development of these complex decision-making skills remain largely unexplored. Here, the educational use of formative script concordance testing (SCT), a written assessment format originally designed to test clinical reasoning, is evaluated in this context through an in-depth qualitative exploration of the learning experiences of participating GP trainees. A ‘think-aloud’ approach to SCT is used, which requires question responses to be justified with a short clinical rationale. Eleven 1st-year GP trainees in Oxford (United Kingdom) completed the formative SCT activity, before taking part in individual semi-structured interviews. Data exploring their learning experiences was transcribed and analysed using reflexive thematic analysis. Thirty-seven codes were generated, and schematically organised into six sub-themes, that provide a descriptive summary of the data and insights into the practical educational utility of the SCT format. Three overarching themes were subsequently produced that relate specifically to aim of exploring in-depth the learning experiences of GP trainees to formative think-aloud SCT: 1) developing complex decision-making skills; 2) opportunity for self-evaluation and awareness; and 3) promoting community and professional identity. The development of complex clinical decision-making and reasoning skills is proposed to occur through guided reflection on performance, facilitated by the think-aloud approach to questioning. Together, this suggests a potential role for think-aloud SCT as a distinctive, structured learning activity to complement trainees’ professional development.

KEYWORDS:

• General practice
• script concordance test
• decision-making
• think-aloud
• clinical uncertainty

Introduction

General practitioners (GPs) are required to make timely and accurate clinical decisions, often in the challenging contexts of complexity and uncertainty [1]. The intricate series of cognitive processes and skills involved, collectively referred to as ‘clinical reasoning’, allow GPs to become experts in assessing and managing undifferentiated symptoms, and identify serious pathology amongst an assortment of common conditions [2,3]. Indeed, clinical errors in primary care are more commonly associated with inappropriate reasoning than insufficient knowledge of the relevant topic [4]. This illustrates the importance of GPs developing these contextualised skills, and their emphasis during postgraduate speciality training [5,6]. It also indicates a role for supplementing trainees’ workplace-based experiences with activities specifically designed to facilitate this learning through repeated practice, personalised feedback and self-reflection [7,8]. To date, the use of such structured approaches in GP training curricula remain limited. The aim of this study is to provide an evaluation of the novel use of a formative think-aloud SCT assessment in this context, through an exploration of the learning experiences and perspectives of UK GP trainees.

The script concordance test (SCT) is a written assessment that aligns its question structure with the principal steps of clinical reasoning [9–11]. SCT first presents candidates with a brief clinical scenario that deliberately retains a degree of uncertainty. Individual questions propose a hypothesis for a related diagnostic, investigatory or management decision (‘If you were thinking of …’), before introducing a novel piece of data (‘and then you find …’). The candidate is then asked to evaluate the effect of this new information on the initial statement (‘this becomes … ’). Responses are given on a five-point scale ranging from −2 (e.g. very unlikely/useless), through 0 (neither more or less likely/useful) to +2 (very likely/useful) (Figure 1.). Scoring is assigned based on responses to each question independently provided by a panel of expert clinicians (typically individuals fully qualified/certified in their speciality) [12]. An example of an SCT question and scoring is provided in the supplementary information available online.

Figure 1. The principle steps of clinical reasoning in alignment with SCT question construction (adapted from Lubarsky et al., 2013) [9].

Originally intended to summatively evaluate clinical reasoning skills, the suitability of SCT in this context has become the subject of debate [11,13–15]. Such limitations of SCT relate to the challenging nature of determining thresholds of competency for these intricate skills, and reliability and validity concerns raised by the use of a graded answer scale and the approach to scoring [16–20]. Despite this, the unique features of SCT have the ability to engage learners in discussion and higher-order cognitive processing of their decision-making behaviours [21–23]. As such, there is increasing acknowledgement of the educational value of SCT as a learning tool used in formative capacities. The deliberate introduction of ambiguity into scenarios and the use of a graded answer scale attempts to avoid a single ‘correct’ and multiple ‘incorrect’ responses to each question [10]. This facilities SCT to function as a written assessment that more closely resembles the subtleties of real-life clinical practice and decision-making between a variety of justifiable options, whilst allowing broad sampling of topics, practical across cohorts and within a short timeframe [9,24]. Recently, the leaning opportunities provided by formative SCT has been shown to be enhanced by adopting a ‘think-aloud’ approach to questioning [22,25,26]. Here, learners are required to justify their Likert scale responses with verbal or written clinical rationales, encouraging reflection on the deeper cognitive processes involved in decision-making in conditions of uncertainty [22–28]. To date, the use of formative think-aloud SCT as a learning tool in postgraduate GP settings is yet to be evaluated. This is despite widespread acknowledgement of the relevance and importance of decision-making skills in the context of uncertainty by clinicians in primary care [29].

The SCT assessment used in this study was delivered by the author to their peer cohort of 1st-year GP trainees enrolled in the Oxford GP training scheme. Access was granted to use a previously validated 145-question primary care SCT assessment, designed by the Department of General Practice, University of Liège [30]. Following translation, 10 scenarios (30 questions) were selected to represent a range of topics across the Royal College of General Practitioners’ (RCGP) training Curriculum [31]. To ensure the expert panels’ answers reflected up-to-date UK primary care practice, responses (including a brief corresponding think-aloud written rationale) were obtained from 13 UK GPs (6 female: 7 male, mean post-qualification experience 13.1 years, across 10 counties in England, Scotland and Wales). Trainees were individually sent an electronic copy of the SCT paper and asked to provide answers with accompanying justifications as an unsupervised, open-book written activity. Their numerical responses were marked according to the scoring approach previously described and returned alongside a copy of the ‘mark scheme’ including the GP panel justifications, and an overall pass mark for the paper (determined using a standard setting approach described by Fournier et al.) [10]. The trainees’ written think-aloud responses were reviewed and considered during a subsequent remote educational session. In small peer groups, allocated scenarios were discussed in depth, including reasons for discrepancies in the expert panel/peer rationales, and techniques that could be used in practice to manage the clinical uncertainties raised. Finally, this was fed back to the whole group in a facilitator-led (GP Training Programme Director) closing debrief.

Methods

This study adopts a ‘big Q’ approach, i.e. utilises both a qualitative research paradigm and methodology, designed to investigate participants’ learning experiences to formative think-aloud SCT through the gathering of non-numerical data [32,33]. The subjectivity of the researcher and their position as an ‘insider’ (i.e. in the same GP training cohort) is therefore considered integral to the overall investigative process [34,35]. GP trainees who completed the SCT activity were invited to participate in the study via email. The study sample size was not specifically determined in advance to achieve ‘data saturation’, however is considered to have generated sufficient information to produce meaningful and practical findings [36,37].

Individual, audio-recorded interviews were conducted by the author remotely using online video conferencing software (Zoom TM). The use of interactive questioning provided a focused, but flexible and non-prescriptive framework to generate rich data relating to each participants’ experience [38,39]. Interviews were guided by a topic schedule, consisting of a series of open-ended questions to explore participants’ learning experiences (see supplementary information available online). The schedule was refined during two initial pilot interviews (included in the analysis), allowing data collection to adapt to emerging findings [40]. Recordings (with corresponding interview notes) were anonymised, stored securely and transcribed verbatim by the author. To ensure accuracy, transcripts were returned to participants for comment and correction (none required) [41].

Data were analysed using qualitative research software (NVivo TM), according to the six phases of Braun and Clarke’s approach to reflexive thematic analysis [34,42,43]. Familiarisation with the data was achieved during transcription and review of audio recordings. Data-driven coding was performed, with individual codes (consisting of phrases, sentences or paragraphs) considered as features in the data capable of conveying a shared underlying meaning when taken out of their original context [36]. These were defined and referenced to facilitate consistency of coding across transcripts (see supplementary information available online). Codes were first schematically analysed into sub-themes to produce a descriptive summary of the data, and facilitate subsequent interpretation and reflexivity during theme generation [42]. Peer debriefing (discussion with an experienced clinical education researcher, see acknowledgements) provided an external perspective through the analytic process [41].

Results

All 11 GP trainees who completed the educational activity consented to be interviewed as part of the study. Participants were in their first year of GP speciality training in Oxford, and had no prior experience of SCT as an assessment method (demographic details presented in Table 1). Interviews (mean duration 41 minutes) yielded a wealth of qualitative data, from which 37 codes were derived. These were grouped into six sub-themes that provide a descriptive overview of the data, and provide practical insights into the educational utility of SCT as an assessment format (Figure 2). From this, 3 overarching themes that specifically relate to the learning experiences and value of engaging with formative think-aloud SCT, from the perspective of GP trainees, were produced.

Table 1. Participant demographics.

Participant	Age	Gender	Prior Experience of SCT	Training Year	Locality
1	28	F	No	1	Oxford (UK)
2	29	F	No	1	Oxford (UK)
3	29	M	No	1	Oxford (UK)
4	34	F	No	1	Oxford (UK)
5	36	M	No	1	Oxford (UK)
6	27	F	No	1	Oxford (UK)
7	37	F	No	1	Oxford (UK)
8	29	F	No	1	Oxford (UK)
9	28	M	No	1	Oxford (UK)
10	33	F	No	1	Oxford (UK)
11	29	F	No	1	Oxford (UK)

Figure 2. Schematic sub-theme analysis of data codes.

Theme 1 – developing complex decision-making skills

The educational value of formative SCT aligns closely with the development of complex primary care decision-making skills. Exposure to their application in the context of uncertainty and across a wide topic coverage, was valued by participants as more authentically reflecting real-life clinical practice (quotes 1–2, see Table 2). The think-aloud approach enhanced the learning opportunities provided by SCT through a number of mechanisms. First, it actively engaged participants in the process of clinical reasoning (quote 3). Providing justifications for their question responses prompted trainees to acknowledge and reflect more deliberately on their underlying thought processing. The ability to then compare and gain feedback from the rationales provided by the expert panel (as well as peers during group discussion) was considered by trainees as being integral to the overall opportunity for learning (quotes 4–5).

Table 2. Participant quotations.

	Quote	Participant
1	‘It stimulates you to think … how do I actually manage that situation, or that consultation with the patient, knowing that I’m not certain?’	5
2	‘Bouncing around from different subjects is a good skill that I don’t practise at the moment, but will have to do as a GP’.	1
3	‘You’re actually having to use the skills that you use in a consultation, where you’re sifting through the information and trying to work out what you think about it. And where there is missing information, you’re trying to fill in those gaps’.	7
4	‘It made me reason why I’d made each decision more than I would have done otherwise’.	11
5	‘I think the whole point is the explanation or the rationale behind [the GP panels’] actions. That’s how you learn, from what their thought processes are’.	10
6	‘You’re exposed to gaps in your knowledge and reminded of certain pathologies … we discussed them in the teaching session, and I looked them up beforehand’.	2
7	‘It was a reassuring process when you thought the same [as the GP panel], and when you didn’t it gave a greater understanding of why they would make those decisions’.	11
8	‘It is still really useful to have a mark … over a span of questions it gives you an overall indication as to whether you’re doing okay’	5
9	‘It’s not very satisfying as a question because you think “what actually is the answer? Why does nobody agree?” ’	8
10	‘Ambiguity is part of real-life clinical practice and we need to learn to deal with it … it’s reassuring that it’s okay to have that feeling of uncertainty’.	6
11	‘It was really interesting to sit and talk through it with other people at the same level as you, bouncing ideas off of each other, getting a feel for what they thought … their justifications and their thought processes’	9
12	‘You’re gaining experience through other people’s experience … learning from other people’s clinical hours’.	10
13	‘I think introducing uncertainty into the mix is a step to independent practice … At some point you start to tentatively develop your own expertise. Maybe we’re not there yet, but at least you have faint glimmers of it coming’.	4
14	‘I thought it was an interesting topic. It was a break from the routine learning of medicine, and just learning about different topics … a nice way to have a bit of critical thinking’	3
15	‘You could tailor it to specific areas of the curriculum, like picking up on red flags or making appropriate referrals or something … or maybe having a few of them up our sleeves to do in teaching sessions from time to time’.	1

Theme 2 – opportunity for self-evaluation and awareness

Formative think-aloud SCT facilitated GP trainees to evaluate their own clinical performance, recognise relevant learning needs, and gain insights into behaviours and characteristics capable of influencing their professional practice. In turn, identifying areas for development prompted self-study, ‘mark scheme’ review and further discussion with peers (quote 6). Furthermore, alignment of their clinical reasoning with the GP panel (as well as peers) was capable of reassuring trainees about their development and progression through training. This was reinforced by receiving a pass mark, indicating a role for standard-setting in the formative assessment (quotes 7–8). The purposeful design of SCT to include clinical uncertainty was identified by trainees, and had the potential to be perceived unfavourably. Interestingly, despite this, trainees reflected on the value of being required to directly acknowledge the issue, contemplate their emotional response and consider strategies to managing it in future practice (quotes 9–10).

Theme 3 – promoting community and professional identity

The SCT activity encouraged trainees to consider their personal identity within, and an appreciation of belonging to, a wider GP community. The think-aloud approach provided a basis for trainees’ clinical thought-processes to be actively sought out, appreciated, and validated in relation to the opinions of others. Participants valued the insights and perspectives offered by their peers and the expert panel, and the opportunity for vicarious learning through exposure to their unique set of experiences and expertise (quotes 11–12). Proficiency in complex and contextual decision-making skills was also recognised by trainees as integral to the identity of a GP, and contributing to the provision of quality patient care. Engaging with, and learning from an activity intended to facilitate this development was perceived as contributing to a sense of legitimate progression through training (quote 13).

Discussion

This study provides the first in-depth qualitative exploration of GP trainees’ learning experiences to formative think-aloud SCT. In doing so, it highlights the potential educational value of its application as a structured activity in speciality training. By requiring the use of clinical reasoning to aid decision-making in the context of uncertainty, trainees are facilitated to develop skills widely recognised as integral to effective primary care clinical practice [2,43]. In addition to the themes outlined above, this study also provides primary care educators with an opportunity to gain insight into the format and innovative application of SCT. Its use as a formative tool in this context is supported by participants widely considering it to be an acceptable, engaging and versatile learning approach (quotes 14–15).

Furthermore, the ability of the SCT assessment design to create meaningful opportunities for learning is consistent with current theories and approaches utilised in postgraduate clinical education [44]. The challenging question structure requires engagement with ‘higher-order’ cognitive processing, as described in the revised Bloom taxonomy [45,46]. Here, these skills are arranged into an overlapping hierarchical, yet recursive structure. By adopting a think-aloud approach, trainees are encouraged to move towards ‘evaluating’ their own decision-making by providing (and/or verbally discussing) their clinical justifications. Each individual SCT scenario is therefore capable of providing a concrete experience from which learning can be derived, and later refined by trainees through application in their clinical practice. This is proposed to occur through a process of guided reflection on performance, as described by Kolb’s model of ’experiential learning’ (Figure 3) [47]. GP trainees consistently highlighted the perceived educational value of comparing their own thought-processing to that of the expert panel and their peers, and gaining feedback through the written think-aloud responses and opportunity for collaborative group discussion. Gibbs’ ‘reflective cycle’ expands on these principles by emphasising the influence of evaluating and analysing the emotional response to an event to generate learning [48]. Here, think-aloud SCT was capable of prompting trainees to acknowledge their feelings around the complexity of decision-making in primary care, especially in relation to clinical ambiguity. Indeed, the issue of dealing with uncertainty in primary care is associated with negative influences on GP trainee wellbeing (e.g. clinician anxiety, concern about bad outcomes) and performance (e.g. reluctancy to disclose uncertainty) [29]. Trainees were provided with an opportunity to recognise and understand their initial ‘gut reaction’ to uncertainty, an important aspect of managing this issue in practice [8].

Figure 3. Kolb’s cycle of ‘experiential learning’ (1984) and think-aloud SCT [47].

There are a few limitations of the study. First, it was designed to provide an in-depth evaluation of the learning experiences of a small cohort of participants, at a single stage of training and from the same locality. The findings of the study therefore inform the practical application of formative think-aloud SCT as a learning activity in a GP training curriculum, as opposed to relating to wider generalsibility. The potential for the author’s pre-existing relationship with participants to influence the views expressed is also acknowledged. Finally, the proposed learning opportunities provided by formative think-aloud SCT are derived from perceived self-development of clinical skills that are inherently difficult to measure. Further work would therefore required to assess how trainees may apply this learning to complex decision-making in their subsequent clinical practice.

Conclusion

This study evaluates the educational value of formative think-aloud SCT through an in-depth qualitative exploration of the perspectives of a cohort of UK GP trainees, and in doing so supports the ability of this approach to create rich and meaningful learning opportunities. Furthermore, it provides educators with practical insights into the application of SCT as a structured tool in postgraduate speciality training. Due to the complexity of developing clinical reasoning and contextualised decision-making skills, this is suggested to complement (as opposed to replace) existing workplace-based learning approaches. It is also hoped this study will act as a springboard to justify the further integration and evaluation of formative think-aloud SCT across additional primary care education settings.

Ethical Considerations

This study involved recruiting GP trainees as participants and implementing an educational activity, with associated ethical considerations. As such, approval was sought and obtained from the authors’ academic institution (The University of Edinburgh Medical Education Research Ethics Committee, ref: 2020/19), and the educational body responsible for the participants’ clinical training (Oxford GP vocational training school). Eligible participants were provided with written information outlining nature of the study and their involvement, alongside their right to withdraw, prior to voluntary recruitment.

Availability of data

Available from the author upon request.

Acknowledgments

The author would like to thank Dr Derek Jones (Clinical Education Programme Director, University of Edinburgh) for his guidance and support as their MSc Clinical Education dissertation supervisor, and the reviewers for their valuable comments on the draft manuscript.

Disclosure statement

No potential conflict of interest was reported by the author.

Supplementary material

Supplemental data for this article can be accessed here.

Additional information

Funding

The author(s) reported there is no funding associated with the work featured in this article.