Assessment of professionalism: A consolidation of current thinking

Abstract

Context: Professionalism has become a hot topic in medical education. Professionalism needs to be assessed if it is to be viewed as both positive and relevant.

Objectives: The assessment of professionalism is an evolving field. This review aims to consolidate current thinking.

Implications: Assessment of professionalism has progressed from an initial focus on the development and attainment of professional identity, through identifying areas of deficiency, to the attainment of a set of identifiable positive attributes and behaviours. It is now beginning to recognise the challenge of assessing a multi-dimensional construct, looking beyond the measurement of behaviour to embrace a diversity of approaches.

Conclusions: Professionalism should be assessed longitudinally. It requires combinations of different approaches, assessing professionalism at individual, interpersonal and societal/institutional levels. Increasing the depth and the quality of reliability and validity of existing programmes in various contexts may be more appropriate than concentrating on developing new instruments. Increasing the number of tests and the number of relevant contexts will increase the reliability of the result. Similarly increasing the number of observers increases reliability. Feedback, encouraging reflection, can promote change in behaviour and identity formation.

Introduction

The medical profession's relationship with society has come under strain in recent years due to a combination of factors, including a reaction to high-profile examples of unprofessional behaviour. The profession has responded by redefining its core values and norms, mainly in terms of character traits and observable behaviours and imposing greater collegiate authority on its members.

The focus on normative definitions of professionalism misses the influence of context, institutions and socio-economic and political concerns and leads to an over-emphasis on codes of behaviour (Martimiankis et al. 2009). Professionalism is a complex, multi-dimensional construct that varies across historical time periods and cultural contexts (Hodges et al. 2011). It has elements at all three levels of House's (1977) PSSP model (Figure 1), which draws on the psychological social psychology, symbolic internationalist and the personality and social structure perspectives, and recognises the relevance of the three levels of analysis and their interactions. For example, a doctor's professional behaviour is dependent not only on their personal characteristics, but may be more influenced by situational and contextual phenomena arising during learning and practice. During this process individuals may look to institutionalised norms and conventions, reproduced and reinforced in day-to-day interactions, to structure their behaviour, giving it meaning and justification. The post-modern view, however, is less deterministic. Individuals are viewed as making sense of institutions through their own unique backgrounds and in the current context in which the institution resides. Meaning is created rather than transmitted and culture is constantly being re-created (Tierney 1997).

Figure 1. House's PSSP model.

Table

Practice points

The assessment of professionalism is an evolving field. Professionalism as a multi-dimensional construct the assessment of which requires a combination of approaches at individual, interpersonal and societal/institutional levels. Assessment should be longitudinal and provide feedback to learners Feedback, encouraging reflection, promotes change in behaviour and identity formation.

Professionalism has, in turn, become a hot topic in medical education with growing recognition of the importance of medical students and doctors developing excellence in professionalism (Stern 2006). The lack of a consensus definition of professionalism has, however, limited its operationalisation (van Mook et al. 2009a). There is little evidence to support the assumption that simply defining outcomes and providing teaching and learning experiences will positively impact on attitudes towards professionalism and/or professional behaviour in medical school or subsequent practice (Jha et al. 2007). Professionalism needs to be assessed if it is to be viewed as both positive and relevant (Cohen 2006; Stern and Papadakis 2006). The assessment of professionalism is an integral component of the GMC's (2009) recommendations for undergraduate curricula and all four domains of its approach to appraisal and re-validation (GMC 2011).

The assessment of professionalism is an evolving field. This review aims to consolidate current thinking.

The scope of assessment

The academic literature on the assessment of professionalism has moved from an initial focus on the development and attainment of professional identity (through identifying areas of deficiency, such as the loss of ethical principles) to the attainment of a set of identifiable positive attributes and behaviours (Baldwin & Daugherty 2006). It is now beginning to recognise the challenge of assessing a multi-dimensional construct and the need to embrace a diversity of approaches.

The primary focus has been on the measurement of professional behaviour, the assumption being behaviour is reflective of the underlying dimensions of professionalism; cognitive, attitudinal, personality and characteristics. However, this has not been supported by evidence from socio-cognitive psychology. For example, attitudes have been found to be poor predictors of behaviour particularly when external constraints, such as social pressure to behave in a particular way, are strong (Rees & Knight 2007). An individual's behaviour is more likely to be influenced by situational and contextual phenomena arising during learning and practice than by their underlying attitudes (Wallace et al 2005; Rees & Knight 2008). This has the potential for students and doctors being unfairly labelled as ‘unprofessional’ in their attitudes by observers who ignore contextual circumstances. The concept of situationally specific professionalism challenges, dilemmas or lapses may therefore be more useful than a global concept of a characteristic or trait of unprofessional behaviour (Hodges et al. 2011). Assessment, therefore, must include the context-dependent nature of professional behaviours. It should include assessment of the decisions, responses and behaviours of all actors in each context, gathering longitudinal data from students and teachers as well as other key players, such as nurses, other health care professionals, patients. It ought to include monitoring of the learning/practice environment and doctors’ interpersonal relationships, e.g. student-teacher, teacher-student, student-patient, for problematic interpersonal phenomena (Holtman 2008). In measuring professionalism symmetry, i.e. where all levels in the organisational hierarchy are evaluated using the same methods, may help alleviate the tension produced by students being assessed by faculty members who do not always practice what they preach (Brainard & Brislen 2007). Inherent in this approach is the provision of feedback to improve the performance of teams as well as improve structural elements, e.g. the habitual, patterned and thus pre-reflexive way of understanding and behaving that helps generate and regulate the practices that make up the social life of the primary care team (Bourdieu 1990).

Similarly, with the emphasis on measurement of behaviours, doctors may be encouraged to ‘fake’ professional behaviours with the potential for doctors with professionally acceptable behaviours, but unprofessional attitudes, being assessed as professional (van Mook et al. 2009a). Assessment methods that capture both behaviours and attitudes require to be further developed and tested, for example observation coupled with conversations during which attitudes are revealed (Rees & Knight 2007). Feedback, particularly where it encourages reflection, may lead to change and promote identity formation (Goldie 2012).

The concentration on the measurement of behaviour has also ignored the knowledge base of professionalism. It is important that students’ possess this knowledge and that it is adequately tested (van Mook et al. 2009b; Hodges et al. 2011).

Professionalism at the societal/institutional level can be understood in the context of the goals, aspirations and collective behaviours of the healthcare and educational institutions and the wider medical profession. Assessment at this level is in its infancy. It is likely to involve measuring, through dialogue and meaningful input from public stakeholders, the extent to which the profession, or one of its subgroups, for example GPs, meet the expectations of the wider society As such, it may take the form of institutional outcomes, e.g. patient outcomes or processes (accreditation requirements) (Hodges et al. 2011). Other approaches, such as critiquing how professionalism has been characterised and the power dynamics of its enforcement, may lead to improved institutional and organisational climate and practice (Hafferty & Castellani 2009).

General assessment principles

Medical competence has long been considered by educators to involve combinations of constructs, which although cannot be observed directly but are measurable, e.g. knowledge, skills, problem solving and attitudes. Constructs were assumed to be stable, generic and independent of other constructs. Many assessment instruments were developed often with the aim of being a single definitive test of a construct, e.g. the MCQ as a test of knowledge. However, the idea of stable and generic constructs has proved no longer tenable (Elstein et al. 1978), and assessment has moved onto competencies. Competencies are tasks that a qualified medical professional should be able to perform successfully. Assessment in the medical education setting has concentrated on measuring individual's performance under test conditions. Assessment should ideally involve measuring performance in everyday practice (van Mook et al. 2009b).

No single instrument can be used for each competency. Combinations of instruments need to be used (van der Vleuten & Schuwirth 2005). This is particularly pertinent when attempting to measure complex multi-dimensional constructs such as professionalism. Instruments that hold context static, e.g. MCQs that measure knowledge base remain valuable in the assessment of competencies.

In establishing the reliability of instruments, traditional psychometric methods require to be extended to defend assessment decisions to the various stakeholders involved. Reliability may not always be conditional on objectivity and standardisation, but often on adequate sampling and the expertise of those making judgements (Schuwirth & van der Vleuten 2006). In considering which instruments to use, the following criteria need to be considered:

• Validity

• Reliability

• Feasibility

• Acceptability

The impact on education and learning is also of importance (van der Vleuten 1996).

The utility of an instrument is a function of the relationship between all these elements. In practice, a trade-off exists between these utility criteria (Thistlethwaite & Spencer 2008). Different weightings need to be applied depending on the context and purpose of the assessment. In high-stake examinations reliability will have higher priority in the choice of assessment method. In formative situations, where the final decision is based on a triangulation of different assessments, reliability can be compromised in favour of educational impact (van der Vleuten 1996).

Assessment tools for measuring professionalism

Building on earlier work by Lynch et al. (2004) and Veloski et al. (2005), Wilkinson et al. (2009) identified nine clusters of assessment tools for measuring professionalism. Box 1 lists the different clusters with examples of assessment tools in each category (see Wilkinson et al. (2009) for individual references). Many of the instruments used have not been fully tested in terms of their reliability and validity (Lynch et al. 2004; Veloski et al. 2005; Jha et al. 2007; Wilkinson et al. 2009).

Box 1 Assessment tools for measuring professionalism

Observed clinical encounters

Mini-CEX

Professionalism Mini-Evaluation Exercise

Standardised Direct Observation assessment Tool

Collated views of co-workers

360 Degree evaluation

Records of incidents of professional lapses

Incident reporting form

Critical incident reports

Simulations

Ethical dilemmas in high-fidelity patient simulations

OSCE

Paper-based tests

Defining Issues Test

Objective Structured Video Examination

Critical Incident report

MCQ

Patient surveys

Patient assessment questionnaire

Simulated patient rating scales

Humanism scale

RCP Patient Questionnaire

Global observer ratings

Global Rating form

University of Michigan Department of Surgery Professionalism Assessment Instrument

EPRO-GP

Amsterdam attitudes and communication scale

Self administered rating scales

Time Management Inquiry Form

Pharmacy Professionalism Instrument

Groningen Reflection Ability Scale

Cross-cultural adaptability inventory

Cultural competence self-assessment questionnaire

Interpersonal Reactivity Index

Penn State College of Medicine Professionalism Questionnaire

In choosing which instruments to use Miller's (1990) framework for the assessment of clinical skills, competence and performance is useful for illustrating their relative position and use (Figure 2). The lowest two levels test aspects of cognitive knowledge while the upper levels focus on behavioural aspects.

Figure 2. Miller's learning pyramid.

Currently the most commonly used instruments are peer assessments, OSCEs, observation by faculty members, which often involve the use of standardised checklists, learner portfolios and critical incident reports. Written comments and reports from formal evaluation sessions, completed by a supervisor and/or other staff are also often used (van Mook et al. 2009b). Increasing the depth and the quality of reliability and validity of existing programmes in various contexts may be more appropriate than concentrating on developing new instruments (Hodges et al. 2011). However, as previously mentioned, methods that capture both behaviours and attitudes and approaches to assessment at societal/institutional levels require further development and testing.

General guidelines for assessing professionalism

General guidelines have been developed for assessing professionalism (van Mook et al. 2009b):

• Professionalism is a multi-dimensional construct and as such should be assessed at individual, interpersonal and societal/institutional levels.

• When measuring professionalism, no single instrument captures all its dimensions. Combining multiple methods, triangulation, is necessary (Thistlethwaite & Spencer 2008). While assessment at performance level is important, knowledge, values and attitudes should also be measured (van Mook et al. 2009b).

• The purpose of assessment must be made clear, particularly whether it is for formative and/or summative purposes. Feedback provided by formative assessment has the potential to change behaviour (Phelan et al. 1993; Papadakis et al. 2001; Goldie 2012), particularly when it is longitudinal, frequent and helps guide remediation (Van Luijk et al. 2000; Projectteam Consortium Abeundi 2005). Using instruments which provide descriptive comments are most effective (Hunt 1992).

• The choice of outcome is related to the usefulness of the method. For example, while peer assessment works well as a formative assessment tool when used summatively, it often fails to discriminate due to a reluctance to judge their peers in a negative light (Arnold et al. 2005).

• Decide what the reference for assessors should be – norm or criteria. With the lack of consensus, concrete, operationalisable definition(s) of professionalism (Stern 2006) criteria referenced standards are preferable. However, setting the reference standard can be difficult as the incidence of professional lapses is often low (Hafferty 2006).

• Increasing the number of tests and the number of relevant contexts will increase the reliability of the result. The closer the assessment is to reality, the more valid it is likely to be (van der Vleuten 1996). Similarly, increasing the number of observers increases reliability. Assessors need to be trained to rate students’ performance objectively and avoid ‘attribution bias’, i.e. the tendency to generalise observed behaviours to all contexts (Stern 2006).

• The assessment should ideally include a situation that involves conflict (Arnold 2002; Hafferty 2006; Stern 2006). Assessment should not only include proposing a solution to the dilemma, but also establish the reasoning behind the proposal (Stern 2006).

• The assessment should incorporate a longitudinal trajectory. Professionalism is a process (Hilton & Slotnik 2005; Hafferty 2006). It should be assessed throughout medical school, post-graduate training and beyond.

• The assessment should be supported by adequate guidance, suggestions for remediation, as well as decisions regarding continuation of training (Hodges et al 2011). There is evidence from retrospective studies that practicing doctors facing disciplinary action from licensing boards had a higher incidence of prior professional lapses (Papadakis et al 2004; Ainsworth & Szauter 2006).

• Feedback, encouraging reflection, should be provided during and directly after the observations (van Mook 2009b). Reflection contributes to individual learning and identity formation (Bravilosky et al. 2011; Goldie 2012). Provision of constructive feedback has been shown to improve professional behaviour (Phelan et al. 1993; Papadakis et al. 2001; Goldie 2012).

• Where assessment tools are to be used in new contexts, re-validation with attention to cultural relevance is important (Hodges et al. 2011).

Declaration of interest: The author reports no conflicts of interest. The author alone is responsible for the content and writing of this article.