Re-examining single-moment-in-time high-stakes examinations in specialist training: A critical narrative review

Abstract

In this critical narrative review, we challenge the belief that single-moment-in-time high-stakes examinations (SMITHSEx) are an essential component of contemporary specialist training. We explore the arguments both for and against SMITHSEx, examine potential alternatives, and discuss the barriers to change.

SMITHSEx are viewed as the “gold standard” assessment of competence but focus excessively on knowledge assessment rather than capturing essential competencies required for safe and competent workplace performance. Contrary to popular belief, regulatory bodies do not mandate SMITHSEx in specialist training. Though acting as significant drivers of learning and professional identity formation, these attributes are not exclusive to SMITHSEx.

Skills such as crisis management, procedural skills, professionalism, communication, collaboration, lifelong learning, reflection on practice, and judgement are often overlooked by SMITHSEx. Their inherent design raises questions about the validity and objectivity of SMITHSEx as a measure of workplace competence. They have a detrimental impact on trainee well-being, contributing to burnout and differential attainment.

Alternatives to SMITHSEx include continuous low-stakes assessments throughout training, ongoing evaluation of competence in the workplace, and competency-based medical education (CBME) concepts. These aim to provide a more comprehensive and context-specific assessment of trainees’ competence while also improving trainee welfare.

Specialist training colleges should evolve from exam providers to holistic education sources. Assessments should emphasise essential practical knowledge over trivia, align with clinical practice, aid learning, and be part of a diverse toolkit. Eliminating SMITHSEx from specialist training will foster a competency-based approach, benefiting future medical professionals’ well-being and success.

Keywords:

• Assessment
• High-stakes examinations
• Competency-based medical education
• Specialist training

Introduction

Single-moment-in-time high-stakes examinations (SMITHSEx) are often considered the gold standard for assessing competence in specialist training. “Single-moment-in-time” refers to how these are conducted on a specific date or within a short period, often at the end of a defined phase of training, for assessing competence up to that point. The high-stakes nature describes how assessment outcomes carry substantial consequences to the individuals involved, with limited retake options in the proximal period. Written and oral formats are primarily used to measure knowledge while objective structured clinical examinations (OSCEs) have been used to assess behavioural competencies such as professionalism and communication (Jefferies et al. 2007; Dwyer et al. 2014), in addition to knowledge and skills. Where specialist training is administered by colleges, their raison d’être is to make a categorical decision on progression from one stage of training to the next. Although not the sole determining factor, SMITHSEx are often seen as the most critical element. Progression decisions are unavoidable and should be based on reliable and defensible information (Cizek 2005). Despite the incorporation of workplace-based assessments (WBAs), minimum case volumes, time-based clinical experience, and other mandatory activities, it is SMITHSEx that ultimately governs progression in training. In the USA, specialty boards employ SMITHSEx to enable specialist certification. Institutions favour their use as they promise to uphold a universally-accepted standard, offer efficiency in decision-making, are highly visible, and provide economies of scale (Heubert and Hauser 1999). However, from the learner’s perspective, SMITHSEx bear little relevance to clinical practice (Craig et al. 2010; Weller et al. 2014; Kahn et al. 2020) and are characterised by inflexibility (Craig et al. 2010; Kahn et al. 2020). A recent editorial argues that there is no evidence that SMITHSEx improve patient safety in specialist training but instead cause unnecessary stress, directing valuable resources towards the goal of passing an exam rather than preparing for independent practice (Thoma et al. 2022).

Practice points

• Single-moment-in-time high-stakes examinations (SMITHSEx) are often considered the gold standard for assessing competence in specialist training, but they are criticised for being inflexible and irrelevant to clinical practice.

• The perceived objectivity of SMITHSEx is not apparent when closely scrutinised. At best, it can claim a form of negotiated or shared subjectivity.

• SMITHSEx focus on knowledge assessment rather than fostering safe and competent clinical practice, driving test preparation rather than genuine learning.

• Alternatives to SMITHSEx, such as programmatic assessment implemented as part of a competency-based medical education (CBME) program, can provide a more comprehensive and contextual measure of competence, and improve trainee welfare.

• Barriers to change include institutional inertia, a perceived threat to professional identity, and financial interests.

We conducted a critical narrative review to re-examine the role of SMITHSEx as an essential component of contemporary specialist training. Specifically, we discuss the commonly cited points both for and against SMITHSEx, explore potential alternatives, and examine the barriers to change. Our intention is not to advocate for the total elimination of written, oral, or clinical assessments from specialist training, but rather to highlight the flaws and unnecessary nature of their application as a single high-stakes event in the present day.

Methods

In keeping with the critical narrative review format (Grant and Booth 2009), the non-comprehensive search sought to identify the most significant items for our topic. PubMed was searched using free-text ([“high-stakes assessment” OR “high-stakes examinations”] AND [“specialist training” OR “postgraduate medical education”]). A similar search was performed in Google Scholar, followed by backward referencing where applicable. Searches were limited to English-language articles published since 2000. Articles focusing on the topics of high-stakes examinations in specialist training and maintenance of competence were included for full-text review. Articles related to the medical school setting or health professions other than medicine were excluded. The goal was to capture discussion points on the use of SMITHSEx in specialist training, including barriers to change and possible alternatives.

We list the arguments for and against the use of SMITHSEx in specialist training in Table 1, acknowledging that this list is by no means exhaustive.

Table 1. Arguments for and against the use of single-moment-in-time high-stakes examinations in specialist training.

For	Against
Familiar to assessors and learners	Inconsistently applied
Able to assess large content areas in a short amount of time	Poor evidence of validity
Provides evidence of accountability to regulators and public	Not explicitly required by regulators
Regarded as an objective measure of competence	Primarily used to assess knowledge, with unnecessary focus on minutiae
Drives learning	Oversimplifies complex nature of competence
Aids in establishing a professional identity	Learners focus on content required for passing at the expense of that required for competent practice
Possible safeguard against role substitution by non-medical health professionals	Emphasis on learning exam technique
Feasible for assessing those with non-recognised qualifications	Immediate recollection of detailed knowledge does not reflect contemporary clinical practice
Financial benefits for administering institutions	Detrimental effect on trainee well-being
	Punitive role in differential attainment
	Financially detrimental to trainees

Retaining the status quo

Accountability to regulators and public

It is often argued that the primary purpose of SMITHSEx is to provide assurance to the public that a high standard of practice exists among specialist doctors (Maudsley 1990). Regulatory bodies responsible for licencing individual specialists and accrediting training programs require evidence of competence but do not dictate the specific means of assessment. Both the General Medical Council (GMC) in the United Kingdom (UK) and the Australian Medical Council (AMC) emphasise multiple forms of assessment within a training program (Specialist Education Accreditation Committee 2015; General Medical Council 2017). While they acknowledge the requirement for summative assessments to demonstrate the achievement of learning outcomes (General Medical Council 2017) and make progression decisions (Specialist Education Accreditation Committee 2015), neither explicitly mandate a single high-stakes event nor dismiss the suitability of multiple low-stakes summative assessments. More recently, the AMC has expressed concern at the reliance on “high-stakes barrier assessments” in specialist training programs, particularly the extent of judgements that are made about trainee competence in single-moment-in-time assessments (Langham 2022).

Some training jurisdictions omit summative assessments altogether. In the United States (US), completing an accredited residency program is the only requirement for practising as a specialist. SMITHSEx at the completion of training are voluntary, with most specialists choosing to become “board-certified” for credentialing and insurance purposes (Fisher and Schloss 2016; Sklar 2016). Danish anaesthesia training does not necessitate SMITHSEx (Ringsted et al. 2003; Yamamoto et al. 2017). Family medicine training in Denmark and the Netherlands does not include SMITHSEx but relies on a range of assessment tools throughout training (Flum et al. 2015). When intensive care medicine training was initially introduced in the UK, SMITHSEx were only a voluntary requirement (Robson 2022).

Ensuring high standards of practice is a reasonable expectation, achievable through a structured training program with robust forms of assessment that measure learning in various domains. While SMITHSEx has traditionally been used to inform progression decisions, alternatives exist within a program of assessment that do not compromise accountability.

Objective measure of competence

SMITHSEx are often regarded as the ultimate objective measure of competence. However, this perceived objectivity diminishes upon closer examination. In classical test theory, a candidate’s score in any assessment comprises their true score and a measurement error component (De Champlain 2010). If an assessment was perfect, the true score would equal the derived score but this rarely happens in practice. Measures of reliability are used to estimate how well derived scores reflect true scores but these measures are sample-dependent, varying based on the candidates being assessed. Previous reviews have shown little published evidence supporting the validity of SMITHSEx for specialist training (Hutchinson et al. 2002).

Even when assessments are guided by marking rubrics, judgments of competence in oral and clinical assessments remain subjective. Inter-rater reliability, which assesses the consistency of measurements across examiners, is also sample-dependent. Multiple-choice questions (MCQs) are often considered highly reliable due to their lack of assessor subjectivity, but they are still subjective as they reflect the value judgements of question creators (ten Cate and Regehr 2019). Standard-setting conversations among examiners involve negotiations on various aspects, further highlighting the subjective nature of assessment. Some have argued that this consensus among a majority of examiners should be considered a form of negotiated or shared subjectivity rather than labelling it as objective (ten Cate and Regehr 2019).

The attempt to standardise outcomes to confer accountability oversimplifies the complex nature of competence (Chin and Lagasse 2017), as competence is context-specific (Bates et al. 2019). A learner’s performance on one assessment or situation is weakly predictive of their performance on a different assessment or situation (Eva 2003). Observed another way, two trainees with the same SMITHSEx score could emerge as entirely different clinicians because they are shaped by their training experience. Truly standardised outcomes may not be achievable (Bates et al. 2019). Some suggest embracing the subjective nature of assessment through the use of entrustment concepts and multiple assessment data points while maintaining fairness as an important goal (Hodges 2013; ten Cate and Regehr 2019).

Assessment drives learning

Despite claims that SMITHSEx enhance learning, they often focus on test preparation rather than fostering competent and safe clinical practice (Bricker 2017). The emphasis on passing SMITHSEx has led to the development of an industry geared towards test preparation, with formal teaching often focusing on content required to pass SMITHSEx rather than content required for competent practice.

The detailed knowledge of scientific minutiae may not be necessary for safe and effective clinical practice. Immediate recollection of detailed knowledge is less important in the digital age when information can be readily accessed. Learning should focus on content that facilitates competent and safe clinical practice, and the ability to seek out specific details when needed. An exception to this may be crisis management, a skill best learnt with regular simulation practice and facilitated by task checklists, though not amenable to being assessed by SMITHSEx.

Studies investigating the association between SMITHSEx achievement and clinical or professional outcomes have shown mixed results. A review of national medical licencing examinations showed no evidence that doctors improved as a consequence of their introduction, with claims of improved outcomes linked to correlation rather than causation (Archer et al. 2016). Trainees who did not pass the Intercollegiate Membership of the Royal College of Surgeons (MRCS) Part B (clinical) SMITHSEx at their first attempt were more likely to receive unsatisfactory competence progression outcomes during training but no association was found with the Part A (written) SMITHSEx and competence progression (Scrimgeour et al. 2018). Similarly, clinical performance scores of final-year residents in one US centre correlated with American Board of Anaesthesiology (ABA) oral and written SMITHSEx scores (Baker et al. 2016). If assessment in the workplace is as good a predictor of competence as SMITHSEx, there is an argument that the latter may be redundant. The written MRCS SMITHSEx showed only a weak positive correlation with performance on the national selection interview for entry into specialist training, though this was stronger for the clinical component of the MRCS SMITHSEx (Scrimgeour et al. 2017). Another study showed that performance in the MRCS SMITHSEx correlated with success in the Fellowship of the Royal College of Surgeons (FRCS) SMITHSEx (Scrimgeour et al. 2019), an indication that people who do well in one exam are likely to do well in another. There is no correlation between MRCS SMITHSEx performance and GMC fitness-to-practice investigations (Ellis et al. 2022b). Conversely, specialists who did not complete maintenance of certification (MOC) SMITHSEx requirements were more likely to be disciplined by US state medical boards compared to counterparts who passed (McDonald et al. 2018; Zhou et al. 2018). This may be interpreted as clinicians who were more likely to obey rules, understood the importance of, and participated in continuing medical education opportunities, were less likely to commit infractions that resulted in disciplinary action. MOC completion among US surgeons, of which SMITHSEx was a component, showed no association with complication rates, with some perceptively suggesting that alternative assessment methods such as video evaluation of technical skills, retraining on state-of-the-art care, or peer review may be more suitable for MOC (Xu et al. 2019).

Establishing a professional identity

Success in SMITHSEx is often perceived as a significant achievement, contributing to professional identity formation. This is not surprising, given the hard work and individual sacrifices involved. In the UK and Canada (but not New Zealand and Australia), specialist college fellowships are generally conferred after SMITHSEx success rather than at the completion of training, implicitly valuing the examination over the whole training program. A shared professional identity can be a positive force and is heavily linked to SMITHSEx success, as people are inclined to identify with the shared investment in training time and cognitive ability (Molleman and Rink 2015). Some have commented that SMITHSEx are intentionally set at a very high standard by specialties to safeguard their standing (Bricker 2017), more so when at risk of role substitution by non-medical health professionals (Fisher and Schloss 2016).

The case for change

One of the primary criticisms of SMITHSEx is that they focus excessively on assessing knowledge. While they may evaluate theoretical understanding and applied knowledge in simulated clinical settings, they often fail to capture essential competencies required for competent workplace performance. For example, in procedural specialties like anaesthesia or orthopaedic surgery, core skills such as tracheal intubation or fracture fixation are better assessed informally in the workplace over an extended period. Additionally, factors like crisis management, procedural skills, professionalism, communication, collaboration, lifelong learning, reflection on practice, and judgement are vital aspects of clinical competence that SMITHSEx often overlook. As a result, SMITHSEx performance alone cannot be considered a reliable indicator of workplace competence; at best, it provides a non-contextual measure of knowledge.

Interestingly, the medical community itself sometimes disregards the importance of SMITHSEx. In subspecialty practice, clinicians such as paediatric anaesthesiologists or orthopaedic spine surgeons are often deemed competent without the need for additional SMITHSEx. The Royal Australasian College of Physicians (RACP) SMITHSEx are a prerequisite for entry to advanced or subspecialty training, with no further SMITHSEx required in areas as varied as cardiology, nephrology, neurology, rheumatology, or gastroenterology (Royal Australasian College of Physicians 2022). In the UK, the Royal Colleges of Physicians introduced SMITHSEx for advanced or subspecialty training only in 2007 (Federation of the Royal Colleges of Physicians 2022). This raises questions about the inconsistency in evaluating competence between different areas of practice. If subspecialists practicing in such diverse areas can be considered competent without SMITHSEx, why do we rely on these examinations for specialist training?

The recent CoVid-19 pandemic further highlighted the incongruence of SMITHSEx, with a number of specialist training colleges altering or suspending certain components of SMITHSEx (ANZCA [@ANZCA], 2020; Langham 2022). This led to a situation where trainees were not formally assessed on certain clinical skills and allowed to progress through training, yet still considered as competent practitioners. This discrepancy underscores the understanding that competence in clinical practice is derived from real-world performance, not solely from SMITHSEx performance. This prompts us to question which competencies are best observed in the workplace rather than through SMITHSEx.

While most countries assume maintenance of competence through continuing professional development (CPD) activities after training, the US implemented mandatory SMITHSEx for recertification every 10 years (American Board of Medical Specialties 2014; Hoyt 2017; Nelson and Butterworth 2018). However, this approach has faced dissatisfaction among specialists (Cook et al. 2016; Hoyt 2017), with some even resorting to legal action (Fisher and Schloss 2016; Hoyt 2017). In response, several specialty boards have introduced alternative assessment formats to replace SMITHSEx. These formats include online open-book assessments (American Board of Internal Medicine 2022; American Board of Surgery 2022), continuous low-stakes assessment throughout the year (American Board of Anesthesiology 2022), and a longitudinal iterative learning assessment (Colson et al. 2019). There is cognitive dissonance in specialist doctors perceiving SMITHSEx as irrelevant to their practice (Cook et al. 2016), a burden (Cook et al. 2016; Howard-McNatt et al. 2019), contributing to burnout (Howard-McNatt et al. 2019), and an avenue for institutions to generate income (Cook et al. 2016), only when it comes to recertification but failing to apply the same logic to initial certification. Good enough for trainees but not for specialists.

The apprenticeship model of medical training allows for ongoing evaluation of competence without relying solely on SMITHSEx. Trainees are constantly assessed based on their performance in the workplace, considering factors such as previous experience, direct observation, and level of training. Supervisors make informed judgments about trainees’ competence, which are not solely dependent on a single examination. This raises doubts about the primacy placed on SMITHSEx and questions the validity of a one-time assessment over the collective judgement of experienced clinicians who have observed a trainee’s clinical practice over an extended period. As specialist training colleges embrace competency-based medical education (CBME) concepts, they cannot simultaneously claim a competency-based system while saying to trainees that “all prior competence assessment does not count if you cannot pass this exam.”

The impact of high-stakes assessments on trainee well-being cannot be ignored. SMITHSEx are often reported as the most stressful aspect of training (Castanelli et al. 2017; Looseley et al. 2019), leading to feelings of hopelessness, anxiety, and shame (Kahn et al. 2020). They adversely affect trainees’ social lives, relationships, mental health, and well-being (Weller et al. 2014). The AMC has expressed concern on the detrimental impact of SMITHSEx on trainees’ lives (Langham 2022). While some specialist training colleges place limits on attempts or restrict access to trainees, other institutions offer SMITHSEx to any registered doctor prior to entry into a training program. In the period from 2002 to 2010, one candidate was allowed 26 attempts in the Membership of the Royal Colleges of Physicians of the United Kingdom (MRCPUK) Part 1 SMITHSEx (McManus and Ludka 2012). Recognising the detrimental effects, the GMC have imposed a limit of six attempts for each part of any national SMITHSEx, effective from January 2022 (General Medical Council 2022). However, the question remains whether these assessments truly reflect an individual’s competence and whether they are worth the toll they take on trainees’ well-being.

Another critical concern associated with SMITHSEx is differential attainment, which refers to variations in performance between different groups of doctors. Differential attainment exists in specialist training SMITHSEx (Esmail and Roberts 2013; Tiffin and Paton 2021; Akinola and Khan 2022; Ellis et al. 2022; 2022a; Luton et al. 2023), and is indicative of broader systemic issues within our healthcare and training systems. Machine-marked MCQs eliminate some forms of bias and while some studies show evidence of examiner bias in clinical SMITHSEx (Esmail and Roberts 2013; Tiffin and Paton 2021), at least one randomised double-blind trial did not (Yeates et al. 2017). There is no evidence that differential attainment is due to learner deficits among ethnic minorities as differences persist after controlling for pre-university achievement, socio-economic status, study habits, living arrangements, and personality (Woolf 2020). It is likely that differential attainment stems from discrimination or implicit bias in the clinical learning environment, primarily affecting ethnic minorities and women. Merely omitting SMITHSEx from training will not solve this problem but we need to acknowledge the significant punitive role SMITHSEx play because of underlying systemic issues.

What are the alternatives?

We have demonstrated that SMITHSEx are not a prerequisite for showcasing accountability to regulating bodies, nor are they a truly objective measure of competence. While they are strongly tied to the formation of professional identity, there is no compelling reason to believe that completing a structured training program without SMITHSEx would diminish the development of a shared professional identity. Any alternatives to SMITHSEx would need to act as an appropriate driver for learning and improve trainee welfare. In our view, programmatic assessment implemented as part of a competency-based medical education (CBME) program, without SMITHSEx, can fulfil these requirements.

Programmatic assessment refers to the systematic and longitudinal acquisition of performance data to inform progression decisions. It involves the accumulation of numerous low-stakes assessments that contribute data points, which are then used to make a high-stakes decision regarding progression (Wilkinson and Tweed 2018). The decision-making process is explicit and transparent, clearly outlining the specific data points used, how they are interpreted, who interprets them, and for what purpose. By shifting from a decision-oriented (pass/fail) SMITHSEx approach to a feedback-orientated model with multiple low-stakes assessments, we move the focus towards providing feedback aimed at improving the quality of care for patients – an aspect that is not currently emphasised in SMITHSEx (Dave et al. 2020). Even though SMITHSEx are unnecessary in a programmatic assessment approach aligned with best-practice guidelines, they often continue to be retained, indicating the challenges faced in implementing change. An early version of programmatic assessment was introduced in Danish anaesthesia specialist training in 2000, which involved direct observation of procedural skills, practice volumes, patient feedback, multi-source feedback from supervising specialists, written assignments, and self-directed learning, all without SMITHSEx (Ringsted et al. 2003). One Canadian emergency medicine residency program has described its recent implementation (Rich et al. 2020). Studies analysing WBAs have shown that a suite of tools can be reliably used to make high-stakes assessment decisions (Hatala et al. 2006; Baker 2011; Moonen-van Loon et al. 2013; Wanderer et al. 2018; Castanelli et al. 2019; Toale et al. 2023). However, it is crucial to strike a balance, as excessive WBA requirements can be a source of stress for learners if poorly implemented. Assessments that are intended to be low-stakes may not be perceived as such by learners unless they are given sufficient agency – that is, their perceived ability to act, control, and make choices in their learning and assessment (Schut et al. 2018).

Eliminating SMITHSEx does not imply the complete removal of written assessments, whether administered on paper or online. Formative written assessments can be easily incorporated into programmatic assessment. Low-stakes formative written examinations throughout training have been shown to predict future performance in SMITHSEx (Kearney et al. 2000; McClintock and Gravlee 2010; Kim et al. 2012; Lee et al. 2012; Jones et al. 2014), suggesting that the latter may be redundant. A systematic review showed the predictive value of low-stakes formative examinations to be stronger for passing SMITHSEx than for failing (McCrary et al. 2021). Another method of conducting written assessments is through progress testing, which involves administering the same test to all learners in a program at or around the same time. These longitudinal low-stakes assessments, typically conducted two to three times a year, cover the entire syllabus. They are designed to discourage learners from solely preparing for a specific test and then “discarding” that knowledge, while also demonstrating knowledge growth throughout the program (Chen et al. 2015; Albanese and Case 2016). At the start of the program, learners are expected to get lower scores compared to their more advanced colleagues, and each test serves a formative purpose. “Failing” a single test does not act as a barrier to progression. One medical school that transitioned from a traditional SMITHSEx model to progress testing found reduced levels of stress among its medical students (Chen et al. 2015). Another showed that growth in learning using progress tests was correlated with performance in a national licencing SMITHSEx (Karay and Schauber 2018). Although more commonly used in undergraduate medical education, progress testing is well-established in specialist training (Dijksterhuis et al. 2013; Noel et al. 2016; Pugh et al. 2016; Rutgers et al. 2018; Halman et al. 2020). While multiple-choice or true/false formats are typically used for ease of marking, other formats such as short-answer questions (Rademakers et al. 2005), OSCEs (Noel et al. 2016; Pugh et al. 2016), and simulation-based assessments (Hall et al. 2020) have been described. For those who value the oral assessment format, formalised case-based discussions can serve as an excellent alternative.

Formative in-training assessments (ITEs) are widely used in US specialty training and share similarities with progress tests. In one example, the ABA administers the same formative ITE annually to all residents, providing performance feedback in the form of a percentile ranking for their training level (American Society of Anesthesiologists 2022). Although not required for progression, residency supervisors may use ITEs to gauge trainees’ progress and preparedness for the ABA SMITHSEx (American Society of Anesthesiologists 2022). When the radiology training program in the Netherlands transitioned from a formative ITE to formal progress testing, they found no difference in the trainees’ performance between the two formats (Rutgers et al. 2020). Similarly, the transition from SMITHSEx to a low-stakes assessment spread out over time within the ABA’s MOC requirements did not result in any notable changes in outcomes in terms of the likelihood of disciplinary action by a state medical board (Zhou et al. 2018, 2019).

Possible barriers preventing change

Institutional inertia stands as a prominent barrier to change. The mindset of adhering to established practices with the notion of “if it ain’t broke, don’t fix it” can impede progress. However, our training programs and assessment methods have continuously evolved based on emerging evidence. We have already transitioned away from long-form essays and unstructured oral assessments, as well as favouring the more effective single-best-answer format over true/false multiple-choice questions. The timing of assessments relative to trainee progression is also subject to adjustment. Embracing change and making bold decisions to alter existing rules and structures are essential for the evolution of assessment and training.

Another significant barrier is the perceived threat to professional identity, often intertwined with survivorship bias. It is difficult for individuals to consider the removal of a process that has become intricately tied to their specialist status. Such a move would require tacit acknowledgement that the requirement may have been unnecessary from the start. As “survivors” of this process, decision-makers possess an inherent bias to retain the status quo. Eliminating SMITHSEx could also be seen as a threat to the professional identity of examiners, who often hold esteemed positions in the specialist training college hierarchy. Notably, conflicts of interest may arise when crucial decisions regarding the role of SMITHSEx are made by current or former examiners.

Financial considerations also pose a barrier to change. Some specialist training colleges limit SMITHSEx to trainees, while others extend its availability to registered doctors more broadly. Certain UK specialist training colleges administer SMITHSEx overseas, predominantly in Asia or Africa. Debates have arisen over the merits of this practice (Devakumar and Mandeville 2011; Muir and Thacker 2011). Critics argue that it undermines local specialist training, provides content that may not be locally relevant, and contributes to the brain drain phenomenon (Devakumar and Mandeville 2011). Conversely, proponents contend that these SMITHSEx fulfil local demands as they are often requested by local partners (Muir and Thacker 2011), particularly for international medical graduates seeking GMC registration and work opportunities in the UK. The scale of this practice is evident; in 2013, the number of overseas candidates sitting the Part 1 MRCP SMITHSEx surpassed those taking the exam in UK for the first time (McAlpine et al. 2014). The revenue generated solely from international candidates undertaking the MRCPUK SMITHSEx could reach a staggering £11.7 million annually (based on current international candidate fees and pre-pandemic candidate numbers; see Table 2) (Royal Colleges of Physicians of the United Kingdom 2019b, p. 1, 2019c, p. 2, 2019a). In 2011, the UK Royal College of Psychiatrists made £600,000 profit from its membership SMITHSEx fees, prompting many trainees to request a refund (Jaques 2013). These financial drivers are not confined to the UK. In a rather brazen act, the American Board of Internal Medicine (ABIM) surreptitiously formed a separate non-profit foundation, transferring US$60 million in assets and purchasing a US$2.3 million luxury condominium with a chauffeur-driven Mercedes S-class car, without informing its physician diplomates (Fisher and Schloss 2016). Meanwhile, ABIM’s own subsequent balance sheet showed it owing $US48 million, balanced by a promise of US$94 million in deferred income from SMITHSEx fees (Fisher and Schloss 2016). American specialty boards have been accused of using employees and subcontractors to publish articles supporting their maintenance of competence programs, inadequately disclosing inherent conflicts of interest (Kempen 2014). Financial incentives make it challenging to envision impartial deliberation on the topic of SMITHSEx within certain specialist training colleges, specialty boards, or among individuals seeking to improve their prospects through migration to higher-income environments.

Table 2. Potential revenue from international candidates sitting the MRCPUK SMITHSEx.

	International candidate fee (2023)	Number of attempts in 2019^a	Potential revenue
Part 1 (written)	£616	7478	£4,606,448
Part 2 (written)	£616	3476	£2,141,216
PACES (clinical)	£1202	4137	£4,972,674

^aFigures are for candidate attempts by non-UK trainees, not number of candidates (Royal Colleges of Physicians of the United Kingdom 2019b, 2019c, 2019a). Candidates self-declared if they were UK trainees or otherwise.

Conclusion

In conclusion, there is a compelling case for re-examining the role of SMITHSEx in assessing competence in specialist training. SMITHSEx are not an essential requirement for demonstrating accountability to regulating bodies nor are they a truly objective measure of competence. Completing a structured training program without SMITHSEx does not necessarily hinder the formation of a shared professional identity. The limitations of SMITHSEx in measuring workplace competence, the inconsistent reliance on these examinations in different areas of practice, the impact on trainee well-being, and the issues of differential attainment all highlight the need for change. While barriers such as institutional inertia, perceived threats to professional identity, and financial considerations may hinder change, they can be surmounted through a collective commitment to evidence-based evolution, unbiased decision-making processes, and a focus on maintaining the integrity of local training programs. By acknowledging and addressing these obstacles, we can pave the way for meaningful advancements in assessment and training practices.

Specialist training colleges need to move past being “exam providers” to “education providers”. Written, oral, and clinical assessments can still exist, but they don’t have to be a single high-stakes event. All assessment should focus on knowledge essential for competent and safe practice, and less on pure factual content or irrelevant minutiae. They should be clinically relevant, reflect practice in the workplace, be used as assessment for learning to identify areas for improvement, and be part of a suite of tools in programmatic assessment. Eliminating SMITHSEx from specialist training will allow evolution from a knowledge-based system to one that is truly competency-based. By doing so, we can better ensure that assessments align with the realities of clinical practice and promote the well-being and success of future medical professionals.

Ethical approval

Not required.

Author contributions

NS and SF individually conceived the ideas contained in this manuscript. NS drafted the initial manuscript, and SF contributed to its critical revision. NS and SF approved the final manuscript for submission and have agreed to be accountable for all aspects of the work.

Disclaimers

None.

Previous presentations

None.

Acknowledgements

None.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

No direct funding was provided.

Notes on contributors

Navdeep S. Sidhu

Navdeep S. Sidhu, MBChB, PGCertHealSc(Resus), FANZCA, MClinEd, FAcadMEd, is an anaesthesiologist at North Shore Hospital, Auckland and Director of Medical Admissions in the School of Medicine, University of Auckland, New Zealand. He is Chair of the Educators Subcommittee, Australian and New Zealand College of Anaesthetists. He has a master’s degree in clinical education and was awarded the 2019 Distinguished Clinical Teacher Award by the University of Auckland. Twitter/X: @DrNavSidhu

Simon Fleming

Simon Fleming, MBBS, FRCS (Tr&Orth), MSc, PhD, MAcadMEd, MFSTEd, MASE, AFHEA, FRSA, is an orthopaedic surgeon, past president of the British Orthopaedic Trainees” Association, past vice-chair of the Academy of Medical Royal Colleges Trainee Doctors Group, and is a co-chair for the International Conference in Residency Education. He has a PhD in medical education and was recognised as the International Leader of the Year by the Royal College of Physicians and Surgeons of Canada. Twitter/X: @OrthopodReg