From theory to practice in learning about healthcare associated infections: Reliable assessment of final year medical students’ ability to reflect

Abstract

Background: Healthcare associated infection affects 9% of patients admitted to hospital. One of the greatest challenges in addressing this problem is transferring theory to practice in relation to hand hygiene. Developing the ability to reflect can promote this transfer. This study illustrates how an instrument to assess the reflective ability of final year medical students was applied to the context of hand hygiene within the infection control cleanliness champion programme (CCP) and demonstrated inter-rater reliability at all three levels of reflection. The results suggest behaviour change in relation to practice. Methods: One hundred and thirty two reflective accounts were used for this study, provided by 44 5th year medical students. Each student had written three reflective accounts for each part of the hand hygiene unit. Results show that the inter-rater agreement was consistently high for all three levels of reflection. However, the least consistent was at reflective level three. Conclusions: These results suggest that the students were able to link theory to practice following the completion of the CCP. It could also indicate that assessors might require more skills and knowledge to enable effective and consistent examination of all areas of reflection.

Introduction

Approximately 9% of patients will acquire an infection when in hospital (SEHD 2004), This equates to 33,000 infections annually, causing approximately 457 deaths. Costs for the NHS amount to approximately £180 million per year (SEHD 2001). Although infection control is multi-modal in nature, it is widely recognised that hand hygiene remains the single most important intervention to prevent and reduce the spread of healthcare associated infections (HAI) (NHS QIS 2005).

The Scottish Executive Health Department (SEHD) published a Ministerial Action Plan on HAI to improve the prevention and control of Healthcare Associated Infection. One of the key strategies identified was to address infection control through multiprofessional training and education (SEHD 2002). However bridging the theory-practice gap in relation to hand hygiene has proven to be one of the major challenges. Studies frequently conclude that despite receiving infection control education, hand hygiene compliance among medical staff particularly remains unacceptably low. It is suggested that effective infection control educational and assessment strategies are developed to ensure appropriate behaviour becomes the norm rather than the exception (Dubbert et al. 1990; Larson et al. 1997; Feather et al. 2000; Hunt et al. 2005).

The cleanliness champion programme (CCP) (NES 2005, NES 2006) is a structured educational programme that prepares healthcare practitioners to maintain a culture in which safety and adherence to standard infection control precautions (SICPs) is of the highest importance. The student is expected to develop a record of their learning journey, incorporating a variety of audits, exercises, reading and self-assessments, which explicitly links theory to practice using written reflective accounts. A major element of the folder of evidence of learning is to demonstrate sound reflective ability. This not only determines knowledge and understanding of the topic, but also enables students to critically analyse past and present clinical practice. It also encourages students to identify ways in which standards can be improved and/or enhanced, and discuss how, as a cleanliness champion this can be taken forward to ensure a positive impact on patient care.

Table

Practice points
•The Cleanliness Champion Programme was incorporated within the 5th year medical students curriculum.
•The students were able to link theory to practice following completion of the CCP.
•Patient care was improved through reflection.
•The tool can be considered reliable for measuring reflection in the context of infection control and hand hygiene.

The hand hygiene unit within the CCP consists of three parts. Part a allows the student to think about the importance of hand hygiene within the healthcare setting, it's significance in the prevention and control of infection, and helps them identify when they should decontaminate their hands during clinical practice. Part b teaches the student about the various kinds of hand decontamination and demonstrates the correct hand decontamination procedure. It encourages the student to reflect on professional behaviour in order to minimize the risk of spreading infection via hands. Finally, part c looks at reasons while healthcare workers fail to decontaminate their hands and encourages the student to reflect on their role as a CC and how they can become an effect role model to others.

The ability to reflect can help reduce the theory-practice gap (Ashley et al. 2006). This is a process of critical thinking and self-assessment to ensure practitioners maximise their learning (Maudlsey & Scrivens 2000; Sobral 2000). Reflection supports health care practitioners in making more sense of both their practice and the uncertain nature of their workplace (Driscol & Teh 2001). The focus of reflective practice in healthcare is to improve patient care through the development and expansion of knowledge, skills and attitudes (Boud & Walker 1993; Klemola & Norris 1997; Moon 1999; Driscoll & Teh 2001). The process of reflection therefore enables healthcare practitioners to re-evaluate and transform their professional practice (Merizow 1990). In other words being unable to reflect can impair the progression to the reconstructive phase of the clinical learning spiral and can compromise required behaviour change in practice (Carr & Carmondy 2006).

However, while the benefits of reflective learning is widely recognized, there are few instruments that have been proven to be consistent in terms of inter- rater reliability for the assessment of students’ reflective abilities (Newble & Swanson 1988; Wong et al. 1995; Boenink et al. 2004).

The purpose of this study was to determine whether acceptable inter-rater reliability was achieved in relation to the assessment of the reflective accounts related to hand hygiene in the cleanliness champion programme. Inter-rater agreement is a familiar concept among social and behavioural scientists to measure the level of agreement between two or more raters or observers (Gwet 2001) state that the key feature and critical components of utilizing a statistical approach is correction for chance agreement. Therefore, without a statistical approach, a risk of overestimation of the agreement level between rates is likely.

Methods

Context

The process of assessment for final year medical students at the University of Dundee includes a portfolio examination. The portfolio includes timed evidence of the students’ clinical experience and their reflections on their progress in relation to the 12 curriculum outcomes (The Scottish Doctor 2000).

Subjects and materials

All 138 final year medical students in 2005/2006 who presented for the final portfolio examination were required to provide evidence of completion of the CCP. Each student was assigned a mentor in which they had to meet with following completion of the programme and who subsequently marked their folder of evidence. All mentors were requested to ask permission to take a copy of each student's reflective accounts during their final meeting. A total of 44 copies were made, therefore 44 students’ reflective accounts were used for the purpose of this study. Verbal consent was obtained to allow the anonymous assessment of all 44. There were three reflective accounts for each student in relation to the hand hygiene units of the programme, which provided us with a total of 132 reflective accounts. Each reflective account was then allocated a number.

Instrument

The reflective ability assessment instrument had been previously developed and used to assess written material of final year medical (Ker 2002). The three levels of reflection assess increasing complexity in relation to the students’ ability to reflect. At level 1, students should describe relevant hand hygiene evidence from their own experience. At level 2, students should evaluate their hand hygiene experience and compare their evidence against hand hygiene standards. Finally, at level 3, students should be able to identify and re-evaluate their own learning needs in relation to hand hygiene. Each hand hygiene reflective account used for this study was assessed using these levels. Each student therefore had a rating for each level.

Each level was assessed as pass, fail or not assessable. Evidence of a good reflector, and those who received a ‘pass’ were students who were able to describe relevant experiences in relation to hand hygiene. They were also able to analyse and evaluate their relevant infection control experiences, which would provide evidence of their progress toward achieving the each outcome of each unit within the programme. Lastly, students were able to re-evaluate their progress in learning about hand hygiene taking into account their experiences in different clinical settings. A poor reflector, and those that subsequently received a ‘fail’ were students who were too generalised in their reflection and related little to their own personal learning and experience. They were too descriptive of hand hygiene activities they had carried out and failed to link their theoretical knowledge with their practice. Non-assessable reflective accounts had minimal or no written details provided.

Raters

Three raters were trained in the use of the instrument in a one-hour workshop.

The trained raters assessed the reflective accounts independently. Two raters reviewed each account:

• raters 1 & 2 assessed the reflective accounts of 14 students (1–14);

• raters 1 & 3 assessed the reflective accounts of 14 students (15–28);

• raters 2 & 3 assessed the reflective accounts of a further 16 students (29–44).

The level of interrater agreement between the two assessors was measured using Cohen's kappa. Although the interpretation of [kappa] is the subject of ongoing debate (von Eye 2006), the most common cited values were used for this study (Viera & Garrett 2005):

Table

kappa <0	Less than chance agreement
kappa 0.01–0.20	Slight agreement
kappa 0.21–0.40	Fair agreement
kappa 0.41–0.60	Moderate agreement
kappa 0.61–0.80	Substantial agreement
kappa 0.81–1.00	Almost perfect agreement

Results

A total of 44 students’ hand hygiene reflective accounts were assessed, with one reflective account for each of the three units. A total of 132 reflective accounts were assessed. Three separate levels of assessment were made on each reflective account, which gave a total of 396 reflective marks.

The rater agreement between all assessors ranged from a ‘moderate agreement’ to a ‘substantial agreement’ and was found to statistically significant (p = 0.000) (Table I). The greatest consistency was found in assessing level one of the assessment (n = 124/132) (94%), and the least consistent was level three (n = 100/132) (76%). Some examples of students’ good reflective account and bad reflective accounts in accordance to the marking criteria are displayed in Table II.

Table I.  Inter-rater reliability of reflective assessments

Markers	kappa	Std Error of kappa	Number of agreements n (%)	Number of disagreements n (%)	Number of non-assessable n (%)
1 & 2*	0.662	0.075	109 (87%)	17 (13)	0
2 & 3*	0.414	0.105	105 (83%)	21 (17%)	0
1 & 3*	0.510	0.089	121 (84%)	19 (13%)	3 (2%)

*p value 0.000.

Table II.  Examples of good and bad hand hygiene reflective accounts

Examples of good reflective accounts	Examples of bad reflective accounts
… This unit allowed me to gain more understanding into the importance of hand hygiene and enabled me to review my own practice and that of others. I have reflected on areas of my practice that requires improvement and listed important steps I have taken to achieve higher standards …	Though something (hand hygiene) that is thoroughly taught at med school, it is interesting to review the information
By completing this unit, I have learned the importance of hand hygiene. From my learning of this module, I have reflected on my practice throughout the years and have identified many times when I should have washed my hands, but didn’t. I have also reviewed the technique I used and altered my behaviour to ensure good practice. I now also see myself as becoming a role model to other staff ….	It has been obvious that I still have to be conscious about maintaining a good hand hygiene standard
From this unit, I learnt how easy it is to overestimate the number of time that hands are decontaminated. I also learnt other reasons for not decontaminating hands which on reflection, I had not considered important. Since undertaking this unit I have changed my practice in that I am more aware of risk assessment and the importance of being a positive role model to others ….	As part of the course on our infection diseases block we carried out a hand washing audit so much of this was nothing new.

Discussion

This work has shown that using a previously validated assessment instrument to measure the reflective ability of 5th year medical students in relation to hand hygiene practice, a group of trained assessors can achieve inter-rater agreement. This is the first time this reflective ability tool has been used in the context of infection control.

In contrast to other studies, this data statistically demonstrated a more favourable result in terms of inter-rater agreement. Pitts et al. (1999) found only a ‘fair’ inter-rater agreement between assessors of portfolios, and Rees & Sheard (2004) failed to demonstrate inter-rater reliability in summative assessment although individual assessors’ judgement was moderate.

Interestingly, although the inter-rater agreement was consistently high for all three levels of reflection, the least consistent was at level three. Girot (1993) believes that this could be due to the fact that some assessors do not possess enough in-depth skills and knowledge to enable effective examination of all areas of reflection. Level 3 is however, not achieved by the majority of students by the time of graduation (Ker 2002), yet it is at this meta-cognitive level that behaviour transformation is most likely to occur (Merizow 1990). The sample size may have affected the results and inconsistency of rating.

Ensuring reliable assessment of students’ work in general continues to challenge medical educators (Epstein 2007). This is widely recognised, especially in terms of reflective assessment, shown by the plethora of assessment tools that have been developed and abandoned over the years due to their lack of validity, reliability and subjectivity (Chambers 1998).

Conclusion

Reflection is a cognitive regulation strategy required for the development of self-regulated learning. It is associated with positive and meaningful learning experience (Sobral 2000), enhancing transfer from theory to practice (Foster & Greenwood 1998; Maudsley & Scriven 2000). This study demonstrated that by ensuring a robust assessment tool for measuring reflection, and ensuring assessors are competent in measuring reflective ability, the tool can be considered reliable for measuring reflection in the context of infection control and hand hygiene. Further review of this sample would help to establish whether changes of behaviour in relation to hand hygiene were sustained in their different clinical practice rotations post graduation.

Acknowledgements

The authors would like to thank Dr Sean McAleer, Senior lecturer in medical education Centre for Medical Education, Taypark House, and University of Dundee for his help and expertise with the statistics.

Additional information

Notes on contributors

E. Burnett

EMMA BURNETT, MSc, BN SPQ IC, RGN is an Infection Control Nurse at NHS Tayside, Ninewells Hospital, Department of Microbiology, Dundee

G. Phillips

Dr GABBY PHILLIPS, MB, ChB, FRCPath is a Consultant Microbiologist and Infection Control Doctor at NHS Tayside, Ninewells Hospital, Department of Microbiology, Dundee

J. S. Ker

Dr JEAN KER, MD FRCGP FRCPE, is the Director of Clinical Skills Centre, University of Dundee, Ninewells Hospital and Medical School. She is a practising general practitioner. Her research interests include clinical skills, the use of simulation and reflective practice.