Can assessment be a barrier to successful professional development?

Abstract

In recent years, there has been a profound shift towards objectivity in high stakes clinical assessments, to reassure the regulators and the public that our graduating clinicians are competent. However, a recent report has suggested that since the introduction of high stakes assessment the relative levels of patient harm have increased. Although, direct causation has not been claimed it is incumbent upon responsible educators to reflect on the possibility that our students are passing the assessments but the assessments are not sophisticated enough to predict real-world competency. In addition, there has been an increasing focus of the student population on seeing the purpose of their clinical education being to pass assessments, rather than to treat patients. This unfortunate focus likely stems from their school education that develops a fixed task focused mindset. In this manuscript, the authors explore the potential issues over mindset and assessment sophistication, and offer possible solutions to tackle both problems.

Keywords:

• Assessment
• Programmatic
• LIFTUPP
• Competency
• Development

Introduction

The need to establish that our health care professionals are competent to undertake the role for which they have been trained and registered for is universally recognised. It is also accepted that to demonstrate competency some form of assessment is needed. Therefore, it stands to reason that the use of competency-based assessment should address this need. However, the stark reality is that recent data suggests that since the introduction of high stakes objective assessments, the harm to patients has not decreased as intended, but in fact increased.^1,2 Cynics may argue that such statistics simply reflect the ‘blame culture’, and can be explained through the increase in patient complaints and ‘ambulance chasing’ legal practices. While, the authors accept that no causal link has been established, it is noteworthy that many of our colleagues who are directly involved with teaching undergraduate students have recognised changes in the ability of many to handle perceived failure, as well as the numerous students with a directive task approach towards learning, exemplified by the eternal question of ‘Is this going to be on the test?’ Therefore, the authors contend that these changes in student behaviour, combined with the failure to alter teaching and assessment practices, could be leading to a situation where students are passing the assessments set, but the teaching and assessments are falling short of developing and predicting real-word clinical competency. To overcome these issues, it is necessary to explore the origin of the student maladaptive behaviours, their impact on learning and the limitations of current approaches to the assessment of competency when undertaken in the current learner behavioural background.

Student behaviours and learning

Self-efficacy is an individual’s belief of their capabilities to exercise control over their own level of functioning,³ and is of critical importance to personal goal setting. The ability to set personal goals is fundamental to personal development^3,4 because the more an individual believes that they can do something, the higher the goal they will set for themselves, and the harder they will try to achieve it.³ Essentially, individuals with a high sense of efficacy visualise success, while those with a low sense of efficacy see only potential failure.⁵ Therefore, the higher the efficacy of an individual the more likely they are to succeed academically, as well as maintain psychological health. It is also important to realise that the development of ‘a resilient sense of efficacy requires experience in overcoming obstacles through perseverant effort. Some difficulties and setbacks in human pursuits serve a useful purpose in teaching that success usually requires sustained effort’.⁵

More recently, concepts around efficacy have been explored through the model of Mindsets.⁶ In this model, two basic types of mindset are described, fixed and growth. The individual with the fixed mindset believes that their intelligence is fixed. This means that if intelligence cannot be expanded through endeavour then any failure is interpreted as a failure of oneself i.e. ‘I am a failure’. This is a destructive thought process, and if present means that every assessment is perceived as an opportunity for failure rather than an aid to future learning. The antithesis of the fixed mindset is the growth mindset. Here, individuals believe that their intelligence can be developed through endeavour and effort in response to feedback. Clearly, within the health profession, it is the growth mindset that is required in the areas related to the intended role, as it is fundamental to the reflective processes that drive changes to self-regulation,⁷ which in-turn underpin the continual strive for improved performance that is essential for patient care.

However, data from a study conducted at the University of Liverpool suggest that 57% of students entering the university may have a fixed mindset.⁸ Although these data did not relate directly to students on medical professional programmes, and further work is needed to establish the size of the problem in this context, it is a worrying finding, especially as the authors would speculate that the problem of fixed mindsets within medical programmes may be even greater as a proportion of the student population, due to the highly competitive nature of the entry into these programmes and the changes to our primary and secondary education systems. Over the last few decades, there has been an ever-growing appetite for standardised assessment outcomes, against which to measure the performance of both children and schools, to such an extent that assessment performance seems to have become the goal of education, rather than its bi-product. This has likely led to many of our children adopting a task focus, as well as potentially being labelled as a ‘success’ or ‘failure’ from a very young age. Many parents have responded by ‘check-boxing’ their children’s lives to ensure that their homework is done to perfection so that they get into the right schools, and ultimately gain a place at the ‘right’ university, doing the ‘right’ degree.⁹ Schools have been subject to league tables, and many teachers have responded by teaching to the test to secure their jobs or to obtain promotion. Personal experience suggests that nowhere is this pressure likely to be more prevalent than upon those children who want to (or have been ‘encouraged’ to) enter the medical professions. Unfortunately, the continuous intervention from well-meaning parents and teachers to ensure that children get the top grades has resulted in students focussing purely on achievements rather than the learning process, and in some cases destroying the enjoyment gained from learning. In addition, data suggest that this over-involvement and micro-management of children, together with unrealistically high expectations, is leading to increased levels of depression amongst young people who are lacking both self-efficacy and internal motivation.¹⁰ Children subject to these types of interventions are more likely to be easily discouraged by failure, develop low efficacy, gain a task focus, have an increased tendency to cheat, avoid difficult situations and ultimately establish a fixed mindset. In this archetype, it is not a case of assessment driving learning, but one of assessment driving behaviour.

Over the last two decades the authors and their colleagues have become all too familiar with the increase in these fixed mindset behaviours of undergraduates and their consequences, which include:

∙	Considering the main purpose of the information delivered in lectures being to gain knowledge to pass assessments, rather than to support the treatment of patients;
∙	Careful task and staff selection to ensure the Work-Based Assessments (WBAs) are passed, rather than them being a mechanism to inform the required personal development;
∙	Challenge to feedback that is perceived as negative, rather than acceptance and endeavour to improve;
∙	Collective efforts to memorise questions and place them into banks to learn ahead of written examinations, rather than using the assessment to gauge what they don’t know or understand;
∙	Organised sessions to mechanistically practice Objective Structured Clinical Examinations (OSCEs), rather than using the feedback they receive to identify areas for deliberate practice4,11;
∙	Regarding portfolios as a series of discrete tasks to complete, rather than an integrated longitudinal tool for personal reflection to drive future development; and,
∙	Seeing patients as commodities from which to gain sufficient numbers of the certain treatments required to graduate, rather than people who need care.12

Basically, in the continuum from high school education, students with a fixed mindset see the staff responsibility as being to teach them what is going to be on the assessment, and their responsibility to be to simply to learn how to pass it. This is a long way from the ideal traits we would like to see in our medical professions i.e. individuals with ‘an enduring inner passion for learning’, who are ‘highly self-motivated, and seek out opportunities and challenges to advance their knowledge and skills’.¹³

Overall, the prevailing fixed mindset in many of our students conditions them to be resistant to our carefully constructed assessment for learning strategies,¹⁴ and all the effort that has gone into planning formative and summative learning opportunities evaporates into the milieu of a series of ‘tasks to pass’, establishing the wrong educational impact.¹⁵ However, being aware of the behavioural issues empowers educators to make change because for health care curricula to cultivate and demonstrate the requisite real-world competencies, they must also establish the growth mindset through having the right educational impact to drive appropriate holistic learner development.

Limitations of current approaches to assess competency

We have come a long way from the Aristotelian view that the assessment of competency simply requires ensuring that an individual possess sufficient knowledge.¹⁶ It is now widely accepted that competency is a multidimensional construct that encompasses not only knowledge and skills but also ‘communication, clinical reasoning, emotions, values, and reflection’.¹⁷ To enable the objective assessment of these dimensions of competency, highly objective tools have been developed, and supported by a plethora of literature, which include: Single Best Answer (SBA)¹⁸; Script Concordance Items¹⁹; Situational Judgement Tests²⁰; OSCEs²¹; WBAs²²; and, reflective portfolios.²³ Although, there can be little doubt that underlying principles for these tools are well grounded, it has been suggested that they may lack sufficient sophistication to measure real-world competency, as well as have the wrong educational impact.^2,12,15,24 The reasons for this include:

∙	An assumption that ‘competency’ is both context independent and stable. This assumption is misguided because competency is neither stable nor context independent.2,12,24–26 Moreover, modern concepts over the meaning of competency recognise that it is the ability to flexibly adapt to changing work circumstances.26,27 Therefore, isolated tests of competency are likely to be inadequate for the prediction of real-world competency.
∙	The outcomes from competency assessments inform the successful students that they are now competent. This signals to the students that any areas of ‘failure’ within the assessment are not important and can be ignored, which is worrying considering that it is likely that there will be a wide range of marks associated with the passing students. Moreover, student success in a competency assessment also indicates to teachers that they can apply less effort in observation and feedback, as the student is now ‘good’.2 In the extreme, a positive outcome from a competency assessment risks putting the student into a state of ‘automation’, which is known to arrest on-going development.4,11
∙	Competency cannot be established through simply measuring experience.28–30 A situation that may undermine many current portfolio or target-based approaches, where the level of activity is used as an important gauge of competence.
∙	Many of the assessment tools lend themselves to psychometric analysis, which unfortunately has too frequently become the wrong driver for assessment design because authenticity (assessment validity) has often been sacrificed for the sake of objectivity to ensure statistical reliability.2,24
∙	The quest for objectivity has also encouraged reductionist approaches to the design of assessment tools that risk their trivialisation by both teachers and students.24 This is a situation that also applies to many reflective portfolios.13

If the fixed mindset task focused behavioural background is superimposed on an intermittent series of ‘assessment packets’, each of which comprises a series of reductionist check-listed items, options lists or tasks that must be passed, then it is easy to see why we have a situation where there may be significant problems with developing holistic real-world clinical abilities of our graduates. In addition, we are also potentially damaging their approach to lifelong learning because students that enter our programmes with a fixed mindset will leave with a more entrenched one, creating a less adaptive clinician.

A move from assessment to continual professional development

Irrespective of any issues with learner’s mindsets, there has been a growing realisation that current approaches to competency assessment within medicine may be too superficial. This had led to calls for change at both the assessment and programme level for an increase in sophistication. At the assessment level innovative suggestions have included linked OSCE stations where errors could be followed across stations,² and ‘entrustability’ where WBAs are true to the holistic trans-domain nature of the real-world task and repeated until the learner is ‘trusted’ to complete the task with minimal supervision.^31,32 While, at the programme level there have been long-standing arguments towards embracing programmatic assessment, where all the assessments are carefully crafted together to form a longitudinal integrated structure to enable a better understanding of how each learning outcome is being developed on an individual student basis.^24,25 This approach also requires a move from a simple numeric interpretation of ‘pass/fail’ to the use of more qualitative data the requires panels of experts to act as an ‘interpretive community’,²⁶ a concept that reintroduces the element of professional judgement into the progression process.

To address the potential lack of sophistication these suggestions all have their attractions, especially programmatic approaches.¹² However, when programmatic approaches have been tried they have been subject to problems with aspects of Utility (a construct that considers assessment in terms of validity, reliability, educational impact, feasibility and acceptability¹⁵), which include: students seeing the proximity of the test as the driver for learning, and not seeing the difference between formative and summative assessments³³; the ability of an assessment element to support or inhibit learning depends on how it is perceived by the individual learner³³; variance in the ability or willingness, of staff to give feedback having a significant impact on learning³⁴; and, staff and student resistance to the perceived increase in workload, combined with difficulty in aggregating data from multiple different assessment formats to enable its meaningful interpretation.³⁴

From the authors perspective there is also the added problem that all of these approaches still suffer from assessment being considered as an ‘event’, and irrespective of how these events are operationalised or integrated, the fact they are events is not constructively aligned³⁵ with developing longitudinally reflective clinicians or with tackling the fixed mindset when present, which suggests that such approaches will still likely foster it. A conclusion that the available data would support for some students.^33,34 Therefore, if the assessment events are likely to be misaligned then the solution would seem to be to effectively remove them by transforming into a model for continuous professional development. In this paradigm, for each student, data within the workplace is continuously collected at high levels of granularity through daily observation by all staff so that any issues on the day can be identified and rectified through ‘coaching’ (telling the student what they need to change next time and how to do it, as opposed to simple feedback – what they did right/wrong this time). This daily data can then be appropriately integrated, triangulated, analysed with the appropriate learning analytics, and displayed in real time to show holistic real-world performance.^31,32 Large data-sets of this type can then be used by students and staff to identify longitudinal trends, which can then be used for personal/guided reflection to inform any need to change self-regulation⁷ through embracing personal deliberate practice^4,11 in combination with supported targeted training, where needed. In addition, the workplace data would be able to be integrated with data from knowledge assessment and simulated situations, which with the right design could also be a daily (or at least regular) event(s) contextually aligned to the requisite clinical activity (Figure 1). In this model of continuous professional development, progression through the programme is measured against developmental goals that are evaluated by an expert panel to be at the point when the learner’s holistic performance across a defined range of capabilities has reached the requisite level of consistency, not simply when a series of isolated assessments are passed. This approach also recognises that different learners will progress at different rates, and that the development of the growth mindset requires endeavour that must be supported and recognised, and not ignored. Furthermore, part of the educational process would be to expect that the learner use the data to develop the essential insight to judge when they themselves are ready, so any panel evaluation becomes a confirmatory process not a ‘breaking news event’. This could be operationalised by the leaner identifying their personal developmental needs ahead of the panel meeting, and the panel confirming/challenging this judgement to establish the degree of insight being shown.

Figure 1 Illustrative diagram of a complete theoretical structure to support continuous professional development. A central database core (A) is populated with the relevant stakeholder outcomes that are aligned in a ‘one to many relationship’. Each of the modules sitting off the data base is fully mapped to it, so that all data can be triangulated through the relevant stakeholder outcomes.

Notes: Daily performance data is collected for each learner in the workplace (B) and aligned in the core to the relevant outcomes. In addition, the work place performance data informs the core of the current activity so that questions from the examination bank (D) can be ‘pushed’ to a student device (C), so that relevant knowledge can be contextually embedded and triangulated with the work-based performance in the core. Furthermore, through having all questions in the examination bank aligned to the stakeholder outcomes, it is possible to link all modalities of assessment to each outcome to form the longitudinal pattern of development (E), combined with a detailed understating of where the issues lie so that the learner can undertake deliberate practice. Through this approach, it is also possible to integrate multiple skills to understand the real-world performance, against the required outcomes, and display this in novel format such as a barcode that shows the periodicity of issues showing the progress towards appropriate independence.

Overall, the authors are describing a model of development where the current assessment focus has been lost in a framework of continuous support informed by large data. The purpose of the approach is to develop a holistic clinician through coaching towards being able to achieve real-world developmental goals that are aligned to a series of capabilities, combined with progression that is evaluated at fixed points in time by both the learner and an expert panel. It is also a model that would hope to overcome the all too familiar problem of staff failing to fail,³⁶ which is a barrier to assessment approaches⁷ because coaching someone how to improve is simply more pleasant than telling someone they are not good enough. However, it must be recognised that what is being suggested is not an all-encompassing panacea for success because it would not work for those students who cannot develop in the realistic time scales, as not everyone is able for whatever reason. Nevertheless, the approach would serve to identify such students sooner, so that they can be guided towards the appropriate path(s) for them.

To operationalise the model of continuous professional development described requires significant innovation in both curriculum design and the technology enhanced approaches to support it (Figure 1). However, relatively advanced designs have already been created and successfully operationalised within our curriculum.^37,38 This has required cooperation between all departments of the school to ensure a unified approach to assessment and feedback so that both generic learning outcomes and those which are subject specific can be assessed within the construct of holistic development so that pertinent feedback is given. At our dental school, which has a non-modular programme we are now able to support the development of our students with the provision of over 5000 observations per student that can be holistically integrated to provide developmental coaching. Crucially, independent data from the UK National Student Survey suggest that there is a high level of student satisfaction (approx. 90%) in the domain of ‘assessment and feedback’ amongst our final-year students, which is remarkable because data suggest that programmatic approaches are not popular with students.^33,34 In addition, this amount of data supports defensible progression decisions because ultimately reliability is a function of data quantity, and validity a function of authenticity.^24,25 To move forward in this area, active research is underway to develop artificial intelligence algorithms to analyse the large data-sets to be able to predict future performance, as well as provide highly tailored coaching advice for every learner. In addition, the changes are subject to on-going evaluation through staff and student feedback, validity and reliability analysis, as well as reaching out to work with postgraduate colleagues to establish readiness for practice through establishing performance. However, this is outside the scope of the current manuscript.

In closing, it is the authors’ opinion that irrespective of what changes are made to learning design, their experience suggests that substantive improvements to student learning can be made by: (1) introducing training early in the programme of study to help affected students overcome their fixed mindsets and prepare them to learn appropriately; (2) identifying and training staff with a fixed mindset, so that they can engage with development rather than assessment; (3) putting regular time aside for staff training over the delivery of coaching, which also needs to be monitored; and, (4) embrace the ethos of holistic continuous professional development to support lifelong learning rather than multiple staged assessment events. This requires having longitudinal approaches to integrate data that transcend the constraints of individual and track across them because real-world skills are integrative and not separate.

Notes on contributors

Luke Dawson is Director of Undergraduate Education at the School of Dentistry and a senior lecturer in Oral Surgery. His current research interests relate to the development and measurement of competency using technology-enhanced approaches.

Kathryn Fox, PhD, is a senior lecturer and teaching lead in Restorative Dentistry at the School of Dentistry. Her current research interests relate to understanding the psycho-social factors that influence learning and the development of professional competence.

Disclosure statement

No potential conflict of interest was reported by the authors.

Acknowledgements

The authors would like to recognise all the staff working in the School of Dentistry at the University of Liverpool for their contributions to developing the approaches outlined in this manuscript.