A literature review of multi-source feedback systems within and without health services, leading to 10 tips for their successful design

Abstract

Multi-source feedback (MSF) has become the accepted mechanism of ensuring the appropriate professional behaviour of doctors. It is part of the mandatory assessment of doctors in training and is to be utilized as part of the revalidation of trained doctors. There is significant variation in the models of MSF currently used within the National Health Service and new models of MSF are being designed by various specialties. No single model has been recognized as the ‘gold standard’. However, there is a large published literature concerning MSF, both in the context of health systems and, more extensively, within industry. This published literature is reviewed, drawing attention to aspects of MSF systems in which there is consensus on effective approaches as well as other aspects in which there is doubt about the optimum approach. In the light of the review 10 principles key in the development of effective MSF models have been produced.

Table

Practice points
•Multi-source feedback has been used extensively in industry and the health sector.
•Multi-source feedback is a valid way to measure professional behaviours in doctors.
•Multi-source feedback can enhance team-working, productivity, good communication and trust.
•Any desired behaviours you wish to assess should be described very clearly.
•The assessment tool should be kept simple, with few items, and fit for purpose.

Introduction

It is has always been implicitly accepted that the medical curriculum extends beyond knowledge and skills to include values and behaviours but now these have been made explicit by the General Medical Council (2001). Assessment drives the curriculum, and the need therefore arises to assess behaviours in the workplace. Multi-source feedback (MSF) (also called 360° assessment) has emerged as the dominant process for assessing professional attitudes and behaviour in the workplace. Extensive literature, both in healthcare and in industry, shows that such assessment can, with certain caveats, be practical, valid and reliable. All described MSF systems broadly follow a model in which a number of colleagues act as assessors of an individual. Their assessments are recorded on a pro forma. The judgements are fed back to the individual, either unadulterated or via an intermediary mentor or supervisor. Systems vary in terms of: number of assessors; method of assessor selection and whether or not the assessor is anonymous to the assessee; the content of the pro forma used in the assessment (areas scored and ratings scale used); and the mechanism for feeding back to the assessee.

The published literature includes both ideas and empirical evidence. Definitive ‘proof’ of the unassailability of MSF as an assessment method is not possible: the observations are by their nature subjective; there is no gold standard; the quality under scrutiny changes with time and final outcomes can only be measured after years, when many other influences would have occurred. Therefore it is important that some rules should be characterized to guide the future development of MSF, based on what is already known. This paper considers a number of key issues that have been distilled from the literature. These relate to the purpose of MSF, validity (content and predictive), the assessors (who, how many and reliability) and training implications. The paper ends by proposing 10 tips to consider when designing a MSF system.

Value of multi-source feedback

When done in the right way for the right purpose, MSF systems have been shown to enhance teamworking (Dominick et al., 1997; Garavan et al., 1997; Druskatt & Wolff, 1999), productivity (Edwards & Ewen, 1996), communication and trust (Waldman & Bowen, 1998). MSF systems are now deeply ensconced in industry: 90% of managers find them helpful (Handy et al., 1996), and almost all top US companies now use them routinely (Atwater & Waldman, 1998; Ghorpade, 2000).

Purposes of multi-source feedback

The literature on MSF in industry provides a useful perspective on its application in healthcare. In particular, there is an important distinction to be made as to the purpose to which the assessment is put. McCarthy & Garavan (2001) provide a comprehensive monograph on the use of MSF in industry. They note that there are potentially five purposes:

• Developing insight into strengths and weaknesses, for the individual (Farh & Dobbins, 1989; Hazucha et al., 1993; Martocchio & Judge, 1997; Fletcher, 1999; Keeping et al., 1999; London et al., 1999; McCarthy & Garavan, 1999) and the organization (London & Beatty, 1993; Lepsinger & Lucia, 1998; Steensma et al., 1998; Tornow & London, 1998) to act upon.

• Enhancing culture change (Cardy & Dobbins, 1994; O’Reilly, 1994; Crossley & Taylor, 1995; Church & Bracken, 1997; Bracken et al., 1998; DeSimone, 1998).

• Summative assessment of performance (Newble et al., 1999).

• Evaluating the potential of individuals, for instance to use in career advice or selection (Tornow, 1993; Lindsey et al., 1997; Facteau et al., 1998).

• Enhancing team effectiveness (by allowing opportunity to comment) (Garavan et al., 1997; Towers-Perrin, 1998).

• Identifying training needs for the system (as opposed to the individual) (Towers-Perrin, 1998).

Purpose and correct identification of low performers

In almost all of the published studies within health systems, the principal purpose of the assessment was the identification of those who may have a problem in the interpersonal domain. Some studies combined this with multi-rater assessment of various cognitive functions (such as problem-solving). Some were directly summative, and some were used to inform the possibility of a problem, such that formative action and/or further assessment could occur. In industry, however, there were often other purposes for the assessments—for instance to help determine outcomes relating to pay or career progress.

The way in which results are used has a critical bearing on whether those identified as failing were truly below standard. If these people suffered a serious disadvantage or setback, then the diagnosis must be certain. However, if the suggested intervention is skilled feedback, followed by intensified scrutiny of both good and bad behaviour, then, at worst, this is embarrassing, annoying, time-consuming and unnecessary. At best, however, it rescues poorer trainees, protects patients, reduces opportunity for disharmony and complaint, mollifies aggrieved staff and provides evidence which may be useful in the future.

It might of course be a matter of concern if not all under-performers were detected. However, the fact is that some were detected in all studies. This means that there is an assessment driver in the system to demand a minimum performance in terms of interpersonal behaviours, which would not be there in the absence of MSF.

Purpose and organizational values

London & Beatty (1993) and Hoffman (1995) report that MSF makes organizational values explicit. This has important practical implications: for instance, even if a small number of descriptors of behaviours were adequately reliable, they may not be sufficiently descriptive of what is desired, if the principle purpose of the assessment is to disseminate these values. This concept is supported by London & Smither (1995).

Insight and self-assessment

Three large studies (Mabe & West, 1982; Fletcher, 1999; Van der Heijden & Nijhof, 2004) have shown that self-ratings tend to be higher than MSF ratings, particularly in the less highly rated individuals. These studies suggest that a potential purpose for MSF is to examine ‘insight’, as measured by this discrepancy between self-rating and ratings by others. Bernardin et al. (1993) point out that managers may not change their behaviour when rated by subordinates. This brings out the inherent irony in MSF systems: A key purpose is to help people with blind spots to develop insight, yet this very lack of insight is the quality that makes acceptance potentially difficult, and so may lead to embitterment rather than change.

Validity of multi-source feedback

Content validity

Carline et al. (1989) described the desirable behaviour of doctors, and rated these using physician associates. Newble et al. (1999) set out a complete new assessment methodology for trainees in Australasia, part of which was ‘physician assessment’, which comprised multi-rater assessment of humanistic behaviours. They chose 12 behaviours to assess, only some of which were ‘humanistic’. Choice of descriptors was based on a major consultation exercise. Zhang et al. (2001) used a similar method to identify humanistic qualities in nurses with a view to future measurement. Ilott & Bunch (1980), on the other hand, used a modified Delphi technique, in a large-scale project, to try to identify the ‘competences’ of surgical SHOs. Wood & Campbell (2004) did the same thing for O&G trainees.

More recent health systems papers tend to quote the GMC's Good Medical Practice (2001) as the arbiter of content validity. However, there is very little evidence as to how broad agreement on content should be translated into ‘domain’ or ‘item’ wording.

Evans et al. (2004) discussed instruments for the peer assessment of physicians. They required such instruments to have data on the way they were developed, or on their validation. They excluded self-assessment tools, those not completed by physician peers and those used only in educational settings. These criteria would have excluded most multi-source feedback tools discussed in this article. They did identify three instruments, all from North America. All these included the assessment of clinical performance and other domains such as communication skills and what they refer to as ‘humanistic qualities’.

Content validity and halo effect

Linn et al. (1975) were not able to find any record of a systematic attempt to devise a reliable and valid assessment instrument for assessment in the affective domain, by peers. They devised a ‘performance rating scale’, comprising 16 items each assessed on a four-point scale. Factor analysis demonstrated that many of the items pertained, at least in part, to a single domain that they called ‘interpersonal or relationship’ factor. This alone accounted for 40% of the variance. A second factor, ‘knowledge or skill’, produced nearly one-third of the variance.

Davidge & Hull (1980) and Dielman et al. (1980) again confirmed by a factor analysis the major influence of just two dominant domains on assessment of undergraduates. Dawson-Saunders & Paiva (1986) showed that only four ‘dimensions’ were enough to encompass the totality of performance during a year of medical student clerkship. Maxim & Dielman (1987), in a large study, rated medical students by house officers and attending teachers in 13 behaviourally anchored rating scales. Factor analysis showed that the large majority of the variance was due to two domains: interpersonal skills and problem-solving. Risucci et al. (1989) used factor analysis to show that ‘interpersonal factors’ explained the large majority of the variance, with ‘knowledge’ once again being the second dominant factor. Ramsey and colleagues, in two studies of MSF of physicians, found by factor analysis that two main qualities, ‘deep cognitive’ and ‘humanistic’, explained the majority of the variance in the ratings (Ramsey et al., 1993, Ramsey et al., 1996).

Whitehouse et al. (2002, 2005) demonstrated that four ‘descriptors’ were enough options to give raters. They showed that with only four domains consistent results could be demonstrated across different raters, and also that individual raters were consistent over time, in rating against these four descriptors. This was confirmed by Wood & Campbell (2004). Principle component analysis again showed that a single ‘communication’ factor accounted for the large majority of the variance.

The emerging message of these papers is highly consistent: despite an intuitive need for semantic declension of performance to a variety of specific competences, just two factors have in reality been the main ones being assessed. One of these is cognitive, which, it could be argued, is better tested by other means. The other is ‘interpersonal’ or ‘humanistic’. The evidence is therefore powerful that an overwhelming ‘halo effect’ occurs in multi-rater assessment in the affective domain, whereby just a single ‘interpersonal’ factor is the main determinant of the outcome.

Predictive validity

No study has yet demonstrated that, over a significant period of time, those trainees identified as the best do indeed outperform their peers in relation to the attributes under question. Keck et al. (1979) demonstrated that a combination of cognitive and non-cognitive factors predict future clinical performance, much better than cognitive factors alone. Dawson-Saunders & Paiva (1986) again highlighted a strong non-cognitive component in performance at medical school and in residency. Whilst only MSF exists to measure such performance it must remain a (very reasonable) assumption that attributes such as ‘interpersonal skills’ are indeed desirable, and do indeed underpin desirable performance. Other tools for measurement of such performance are needed. Church (2000) however, shows that in industry, where measurement of performance is often better developed, better (but not perfect) evidence exists for the predictive validity of MSF.

The assessors in multi-source feedback

Non-medical assessors

Risucci et al. (1989) helped establish the role of non-traditional assessors in the workplace, in the assessment of humanistic, interpersonal qualities. He compared peer assessment and supervisor assessment of surgical trainees. Both groups of assessors agreed remarkably with regard to overall rating across the 10 categories while differing in their emphasis.

Butterfield & Pearson (1990) moved the debate on from peer assessment of residents to assessment by nurses of doctors’ ‘humanistic behaviour’. They showed differences in perception as to the definitions of desirable humanistic behaviours, which understandably created some dissent as to whether nurses were qualified to assess doctors—albeit in non-clinical domains. Undeterred, Butterfield & Mazzaferri (1991) developed a form for nurses to assess house staff in relation to humanistic behaviour. In a large study, he was able to show that the instrument was both practical and reliable. They pointed out, however, that the nurses’ picture was often a different perspective from the medical one.

Wenrich et al. (1993) reported the substantial transition to the use of nurses in the assessment of doctors. The study agreed with Butterfield in showing feasibility, although residents again did not universally like being assessed by non-doctors. As with Butterfield, it also showed a moderate correlation between assessment by non-doctors and doctors, once more raising the question as to which perspective was to be taken more into account. Ramsey et al. (1993) importantly provided evidence that the assessee could nominate his/her own assessor, without biasing the outcome.

Number of assessors

Clearly, the issue of numbers of assessors (‘raters’) relates in part to potential for rater error or bias. The fact that rater error can occur has been discussed by Saavedra & Kwun (1993). Bettenhausen & Fedor (1997) point out that in peer rating there is a tendency to leniency, so as to minimize bad feeling. However, multi-rater assessment takes these issues into account by virtue of its nature, in not relying on one source. Thus the issue comes down to how many raters are needed to minimize the chance of an inaccurate rating.

Butterfield & Mazzaferri (1991) estimated that only five or six raters could provide a representative picture for each trainee. Wenrich et al.'s 1993 study calculated that 10 to 15 nurses were needed. Ramsey et al. (1993) suggested 11 raters. Woolliscroft et al. (1994) had more disappointing results. They agreed that only five to 10 Programme Directors would be needed to get a representative result, when rating interns for ‘humanistic qualities’. However, the figure for nurses was 10 to 20, and for patients, more than 50.

Ramsey and colleagues (1996) analysed 3005 questionnaires relating to MSF of 187 physicians. Ten to 11 responses per physician were necessary to achieve a generalizability coefficient of 0.7. Wood & Campbell (2004) calculated the need for eight raters of O&G trainees to give an intraclass coefficient of 0.8.

There might be many reasons why any of these figures differ. First, for some studies, more than just ‘interpersonal’ aspects were being measured. Second, it could be that some groups of raters have a tendency to be inconsistent in their rating. They may have had different training, milieu or natural propensities. And different roles might have different experiences of the assessee. Third, expansion of categories to be rated could conceivably add subjectivity, or irrelevancy, or inadequacy of the rater to be in a position to judge. Finally, raters may have differed in the amount of training they received. This opens the possibility that the number of raters might be susceptible to further reduction, by virtue of better, more focused assessment instruments, and better trained raters.

Accepting the assessments

If the systems could be well accepted and motivating for assessees, then the worry about the lack of consistent evidence on driving improvement would be dissipated. One might reasonably assume, in that case, that if assessees liked the system and found it motivating, then organizational benefits would ensue. Evidence for this is abundant (McEvoy & Buller, 1987; Fedor & Bettenhausen, 1989; Riggio & Cole, 1992; Tornow, 1993; Yukl & Lepsinger, 1995; Maurer & Tarullli, 1996; Towers-Perrin, 1998; Wimer & Nowack, 1998). Additionally, there would be an upward spiral, by virtue of using the system to communicate the values and expectations of the organization in relation to professional behaviours. The Alimo-Metcalfe study (1998) makes the interesting point that MSF systems may highlight strengths of which the assessee is unaware.

Fedor & Bennenhausen (1989) and others (Becker & Klimoski, 1989; Bettenhausen & Fedor, 1997; Bilsbury, 1998), say that it is therefore critical to success to stress the developmental nature of the system to assessees. Barclay & Harland (1995) add that the trainees should perceive their raters as competent, and that they should have the opportunity to have potential errors or biases re-examined before they go on the final record.

Training

Kaplan (1993) notes that receiving negative feedback can demotivate. Industry provides some important insights into this field. McCarthy & Garavan (2001) have reviewed the literature extensively, and feel that training of both raters and assessees is important. They feel that raters need to understand about the common forms of bias, such as halo effect and centralization (Hoffman, 1995).

Many authors agree that training in giving feedback, however, is done badly, to the great detriment of the system (Holmes and McCall, 1997; Lindsey et al., 1997; Kanouse, 1998). If managers are not trained in giving negative feedback, they may hate doing it, and so reject the MSF system (Folger & Cropanzano, 1998). Such problems characteristically occur in hierarchical organizations (London et al., 1990; Murphy & Cleveland, 1991). This can create tension between raters and assesses (Hautaluoma et al., 1992). Interestingly, however, not much can be found in the literature concerning the training of assessees to receive feedback.

Use of computerized systems

There is little evidence in the literature concerning this topic but what there is, is encouraging. Penny (2003) compared pen-and-paper with electronic systems, and concluded that the method did not influence the consistency of raters. Archer et al. (2005) described a successful system in which assessments were centrally scanned into a computer for summarizing and analysis. Payne (1998), however, warned that people may have an unreasonable respect for electronic data and numerical scores, and that a danger of electronic systems might be that they portray subjective data as objective.

Conclusion

It has become clear that communication and relationships need to be part of the formal medical curriculum, and that ‘humanistic’ behaviour needs to be assessed. Published studies, within and without health systems, to a greater or lesser extent, bear out the themes of validity and usefulness of multi-rater assessment in the workplace. The complete lack of a published opposing view may be due to publication bias, or to the difficulty in demonstrating such a conclusion. However, given the three decades of literature, it is most likely to be because multi-rater assessment is indeed a valid and potentially useful tool. Careful planning needs to go into the design of an MSF system to ensure that those being rated predominantly welcome the system. From a review of world literature in both industry and health, it is possible to postulate a set of rules for the future development of MSF systems.

Ten tips in the development of multi-source feedback systems

Develop a positive culture

A positive and supportive ‘culture’ or ‘climate’ of the organization will help motivation to change behaviour, but use of MSF in a negative culture can demotivate. Organizations should use the introduction of MSF as a time to address ‘climate’ issues.

Be clear about the purpose

There is a vital distinction between the developmental use of MSF, and its use in decisions that affect careers. For the latter purpose, the system must be well proven, mature and owned by all.

Clearly express any desired behaviours

Use of MSF disseminates an understanding of desired behaviours, and so the opportunity should be taken to make desired behaviours clear and explicit.

Keep the number of items to be scored few

Whatever the wording of items, assessments of behaviours in the affective domain are dominated by a single ‘halo effect’ perception of the person's interpersonal skills. Large numbers of items do not add discrimination, and may increase inconsistency of rating.

Keep the scale simple and fit-for-purpose

The construction of the MSF scale, and the addition of any option for free text, should reflect the purpose of the assessment.

Use six to 10 raters

The better the instrument, the more generalizable the results from just a few raters—but sampling multiple sources is an inherent element of the assessment.

Compare results with self-assessment

Assessees tend to be more lenient with themselves than raters are. A large discrepancy between the two suggests lack of insight. Discrepancy is reduced by the assessee having a clearer grasp of the desired behaviours. The implication is that MSF should be compared with self-assessment. Furthermore, the assessee's view may add an important and valid perspective, which was glossed over by the raters.

Train those giving feedback

The power of MSF to change behaviour depends on the quality of the feedback. MSF systems demand training of those giving feedback. Skilled, unthreatening feedback is particularly important when the assessee is learning new behaviours.

Skilled feedback needs to be backed up by educational planning, which itself implies time and training to fulfil educational goals.

Involve the assessees

Assessee involvement in the development and implementation of multi-source feedback is likely to enhance the quality of the system and maximize its potential.

Incorporate development

MSF is an opportunity to develop shared understanding in an organization. Raters and assessees should help develop the system together, and should be trained together in dealing with improving performance, in areas shown to be weak. Investigators’ initial attempts at wording will result in some domains being less consistently interpreted than others. There should be an ongoing dialogue between all parties in the organization, to help standardize understanding, and to help develop future iterations of the system.

Additional information

Notes on contributors

Laurence Wood

LAURENCE WOOD is an associate postgraduate dean for education in the West Midlands Deanery and a consultant in obstetrics and gynaecology at University Hospital Coventry and Warwickshire in Coventry.

Andrew Hassell

ANDREW HASSELL is an associate postgraduate dean for education in the West Midlands Deanery and a consultant in rheumatology at the University Hospital of North Staffordshire in Stoke on Trent.

Andrew Whitehouse

ANDREW WHITEHOUSE is a director of hospital and specialist education in the West Midlands Deanery, and a consultant physician at George Eliot Hospital in Nuneaton, Warwickshire.

Alison Bullock

ALISON BULLOCK is Reader in Medical and Dental Education in the School of Education at the University of Birmingham and a senior member of the Centre for Research in Medical and Dental Education in the School of Education.

David Wall

DAVID WALL is a deputy regional postgraduate dean in the West Midlands Deanery and a professor of medical education at Staffordshire University.